AuriStor File System Client Installers

RedHat Enterprise, CentOS, and Fedora Linux Repository installer

Release Notes

This AuriStorFS repository installer supports will install a rpm repository that provides kernel modules for:

Red Hat Enterprise Linux 6, 7, 8, 9 and 10
AlmaLinux 8, 9 and 10
Rocky Linux 8, 9 and 10
Oracle Linux 8, 9 and 10
TuxCare EL 7.9, 9.2 and 9.6
CentOS 7 and 8
Fedora 41 and 42
Amazon Linux 2

For Debian and Ubuntu client please read Updated AuriStor Client support for Debian and Ubuntu.

Installation Instructions

yum install auristor-repo-recommended-10-1.noarch.rpm
yum install yfs-client
edit /etc/yfs/yfs-client.conf to specify the cell name
chkconfig yfs-client on
reboot

All RPMs with the exception of Fedora 41, Fedora 42, and Red Hat Enterprise Linux 10 are signed with RPM-GPG-KEY-YFS

Fedora 41, Fedora 42, and Red Hat Enterprise Linux 10 RPMs are signed with RPM-GPG-KEY-AuriStor-2023

GPG Signing Key expired on 2 May 2025

On 2 May 2025 the AuriStor GPG certificate used to sign Fedora rpms expired. The auristor-repo-recommended-10-1.noarch.rpm installs the appropriate GPG signing key for each distribution.

New v2021.05-64 (22 Aug 2025)

New Platform Support
- Red Hat Enterprise Linux 10 [x86_64 and aarch64]
- Red Hat Enterprise Linux 10 Real-time [x86_64]
- Red Hat Enterprise Linux 9.6 [x86_64 and aarch64]
- Red Hat Enterprise Linux 9.6 Real-time [x86_64]
- Alma Linux 10 [x86_64 and aarch64]
- Alma Linux 9.6 [x86_64 and aarch64]
- Oracle Linux 10 [x86_64 and aarch64]
- Oracle Linux 9.6 [x86_64 and aarch64]
- Rocky Linux 10 [x86_64 and aarch64]
- Rocky Linux 9.6 [x86_64 and aarch64]
- TuxCare EL 7.9 ELS [x86_64]
- TuxCare EL 9.2 ESU [x86_64 and aarch64]
- TuxCare EL 9.6 ESU [x86_64 and aarch64]
- Debian 13 "Trixie" (Linux 6.12 LTS kernel) [x86_64 and aarch64]
- Linux 6.16 and 6.17 kernels
- Linux RISC-V (built upon request)
The v2021.05-64 release fixes a race condition which can lead to a kernel panic when executing the "unlog" command.
Linux kernels 3.19 and later (all supported distributions other than el5, el6, or el7) replaces the Linux dentry lifecycle management with a model which is much closer to that used by in-tree filesystem modules. These changes address the following issues
- It is now possible to " mv /afs/example.org/dir1/file1/afs/.@mount/example.org/root.cell/dir1/file1" without deadlocking the kernel.
- The dentry invalidation race with rename operations which resulting in a kernel deadlock has been eliminated.
For all Linux kernels, an afs mountpoint dentry might refer to either the mountpoint inode or the target volume root inode. Beginning with this release, the bouncing between a mountpoint inode and a volume root endpoint is prevented.

v2021.05-63 (31 May 2025)

New platform support:
- Red Hat Enterprise Linux 10 and AlmaLinux 10
- Linux 6.16 kernel
When AuriStorFS cache managers added support for per-file ACLs in v0.1, the masking of access rights based upon the inode's S_IRWXU mode bits were skipped when reading objects from AuriStorFS fileservers. The masking continued to be applied when reading objects from OpenAFS fileservers. This release applies S_IRWXU mode bits masking of granted access rights to both AuriStorFS and OpenAFS fileservers.
The existing fake stat mountpoint evaluation logic assumes that the fileserver returned a AFSFetchStatus.ParentVnode that refers to the parent directory from which the mountpoint was located. However, this is only guaranteed to be true if the mountpoint does not have a per-file ACL (which is permitted) and has never been cross-directory hard linked (which is also permitted).
This release improves the error handling logic to fail the fake stat evaluation if the AFSFetchStatus.ParentVnode does not match expectations.
This release fixes a metadata inconsistency when a mountpoint is created when disconnected mode is enabled. A mountpoint should be reported to the vfs as a directory and not a symlink.
The cache manager must track for each inode whether the object's status infomration was received from an AFS3 fileserver or an AuriStorFS fileserver. This release alters when that information is recorded to ensure that it is recorded each time the inode's status information is updated by a fileserver response.
This knowledge is required to properly determine how to interpret the status information.
The NFS exporter logic used questionable logic when resolving the parent inode of an afs volume root directory inode. This release implements a much more reliable mechanism.

v2021.05-62 (24 April 2025)

New platform support
- Fedora 42
- Linux 6.15 kernels
This release includes yet another change to d_revalidate() processing to handle the case in which the dentry->d_inode is the volume root directory inode and the volume has been deleted. Previously, an attempt to "ls" the directory contents would have succeeded with the output obtained from the dentry cache. With this change, the "ls" will fail.
v2021.05-60 added an unresolvable dependency on "kmod" to rhel5 and rhel6. The correct dependency, "module-init-tools", has been added in this release.
Binaries built from the rpm spec file will now include the full version number instead of just the base. "2021.05-62" instead of "2021.05".
This release improves consistency of vnode metadata interpretation when the status information is received via an RPC which does not return a callback.

v2021.05-61 (31 March 2025)

The Red Hat filesystem package installs the /afs directory on Enterprise Linux and Fedora. Updates to the filesystem package fail when the AuriStorFS client is running and the "auristorfs" filesystem has been mounted on /afs. The failure occurs because the filesystem package expects the /afs directory to have the following attributes: (root), (root), 0555 whereas the AuriStorFS dynamic root directory advertises (root), (root), 0755 and is readonly. If the values differ, then rpm attempts to recreate the /afs directory which fails.
This release alters the dynamic root directory mode to match the expectations of the filesystem package. This change will not correct the behavior for systems on which [afsd] dynroot is disabled.
Attempts to bind mount a path in /afs can fail when a source path contains a path component that was earlier judged invalid during Linux vfs d_revalidate. The failure occurs during the move_mount syscall when d_set_mounted() walks the source inode's dentry parent chain. If any unhashed dentry is found during the walk the move_mount syscall fails with ENOENT.
Linux does not permit a directory to have more than one parent. However, more than one AFS mountpoint can refer to the same volume root directory. When using AuriStorFS a rw volume without a .readonly snapshot is accessible by a minimum of three paths
1. /afs/your-cell-name.com/.../volume
2. /afs/.your-cell-name.com/.../volume
3. /afs/.@mount/your-cell-name.com/volume
Since a directory is only permitted to have a single parent, the AuriStorFS cache manager identifies one of the paths as the canonical parent. The bind mount failure occurs when the canonical dentry is valid but one of the ancestors of that dentry are not. This release introduces an ancestor validation check to prevent d_set_mounted() from finding an unhashed dentry when it walks the parent chain.
Linux kernel v6.10-rc7 and later removed functionality which broke AuriStorFS reads of mapped multi-page folios. This release adjusts for the loss of functionality which could result in an infinite loop.
At startup, log to the console the size of various hash tables and the number of elements that are expected to be stored in them.
yfs: Allocating 32768 vcache hash table buckets to store 20000 entries
yfs: Allocating 8192 dv/dc hash table buckets to store 10000 entries
Instead of allocating the same sized hash table for the vcache and dcache objects regardless of how many elements are expected to be stored, scale the hash table sizes. The vcache hash table will scale from 4096 buckets to 131072 buckets with a target of 4 entries per bucket. The dcache hash tables will scale from 4096 buckets to 65536 buckets with a target of 2 entries per bucket.
Historically, afs cache managers which create symlinks (including mountpoints) locally allocate a fake infinite lifetime callback to permit applications to use the symlink (or mountpoint) without requiring FetchStatus RPCs to be issued to the fileserver. This approach has drawbacks because it means that fileservers will not issue a callback break when the volume containing the symlink is taken offline, moved, deleted, etc. It also means that a callback will not be sent if another cache manager removes the symlink.
Beginning with this release the AuriStorFS cache manager will no longer allocate a fake infinite lifetime callback and will instead rely upon fileserver allocated callbacks.
"fs makemount" is supposed to create a mountpoint even if the target volume does not exist or is inaccessible. v2021.05 introduced support for creating mountpoints which kafs which broke the ability to create mountpoints to non-existent volumes.

v2021.05-60 (6 March 2025)

Linux maintains a directory entry (dentry) cache containing the results of names that exist and those that do not. Each time a cache hit occurs Linux queries the filesystem to validate whether the cache entry is valid or not. An invalid dentry is unhashed but might still be linked to in-use inodes.
When revalidating a dentry it is critical that a dentry only be invalidated if there is 100% confident that the dentry is invalid. An invalidated dentry linked to an in-use inode can prevent successful creation of a bind mount.
This release includes many changes to reduce the risk that a dentry might be invalidated due to a transient error during dentry revalidation. The internal use of ENOENT and ENODEV has been reduced. In addition, raw afs error codes such as VNOVOL, VMOVED, VL_NOENT and others are no longer accepted as an indication that that the dentry is invalid. The true meaning of such errors cannot be determined outside of the context in which they were received.
Recent reports of bind mount failures have been accompanied by "yfs: rehashed dentry for cell xxx vnode x.y.z (name: nnn)" warnings logged to the console (dmesg) but no warning messages indicating while the dentry was invalidated. A rehashed dentry is one that was previously invalidated but is in fact still valid.
This release introduces new warnings when revalidating either a MOUNTPOINT or VOLUMEROOT and one of the following occurs:
- a vnode lookup returns one of ENOENT, ESTALE, or ENODEV
- a vnode lookup succeeds but the inode has been replaced
- an attempt to fetch vnode status fails and the request error state evaluates to either VNOVNODE or ENODEV.
The cache manager retries fileserver RPCs up to 100 times when the fileserver replies with VBUSY, VOFFLINE or VSALVAGE. The cache manager fails the application request after the limit is reached. Prior to this release, the failure would result in the cache manager returning an ENODEV error to the application. But ENODEV indicates that the required volume does not exist and the fileserver has not indicated that the volume does not exist. The cache manager will now return EWOULDBLOCK to the application because the volume might be temporarily inaccessible.
Many OpenAFS and AuriStorFS fileservers can return VNOVOL as a side effect of a race between a volume going offline so that control can be given to the Volume Management Service (volserver) and an incoming RPC requesting that the volume be attached. A VNOVOL can also be returned because the volume was temporarily deleted from the vice partition prior to being replaced by a new clone, or prior to the fileserver marking the volume group as having been moved to another fileserver.
The fileserver replying to an RPC with VNOVOL cannot be trusted to mean that the volume has been permanently removed unless it is matched by the Location Service reporting the Volume Location (VL) entry has been deleted. From now on when a VNOVOL response is received and the VL entry exists, the cache manager will return EWOULDBLOCK instead of ENODEV to the application.
Prior to this release, any VL error code returned from the Location Service would be interpreted as meaning that the requested VL entry does not exist. Starting with this release only the VL_NOENT error will be interpreted as the VL entry does not exist.
Restrict the internal use of the ENOENT error code so the only use of ENOENT is to indicate that a directory entry does not exist. Previously, ENOENT could be returned if a disk cache entry could not be allocated or found. A disk cache related ENOENT returned when processing a directory might have been confused for the requested directory entry not existing.
Restrict the internal use of the ENODEV error code so that the only meaning of ENODEV is that the required volume does not exist. Previously, ENODEV could be returned due to transient errors when failing to contact the Location Service.

v2021.05-59 (13 February 2025)

The fix for garbage collection of idle unixuser objects introduced in v2021.05-56 was incomplete. As a result, Rx client connections might not be garbage collected.

v2021.05-57 (9 February 2025)

v2021.05-56 kernel modules for Linux v4.10 and earlier kernels would not load. This includes el7, el6, el5 and Debian 8.

v2021.05-56 (7 February 2025)

Minor bug fixes to the Rx RPC implementation
The fix for garbage collection of idle unixuser objects introduced in v2021.05-55 was incomplete. As a result, Rx client connections might not be garbage collected.
Linux 6.14 kernel support

v2021.05-55 (23 January 2025)

Minor bug fixes to the Rx RPC implementation
During garbage collection of idle unixuser objects, any Rx client connections associated with the unixuser (uid or pag) that were in use by another thread would not have their cache manager reference dropped. As a result the Rx client connections could never be garbage collected which is problematic on systems which churn through many unixusers. The behavior has been changed so that unixusers objects with in use Rx connections will not be freed. This permits future attempts to garbage collect unixusers to do so only after the Rx connections can also be freed.
Although this misbehavior has been present for decades, it is only since v2021.05-52 that the problem has been observed. It is possible that the changes to reduce the holding of the GLOCK has created a window of opportunity for connections to be simultaneously in use during unixuser garbage collection.

v2021.05-54 (19 January 2025)

The UNIX cache manager changes continue AuriStor's efforts to address a report that was received at the end of September. The end user site uses an OpenAFS cell to distribute software for execution by IBM Spectrum LSF (load sharing facility) managed compute nodes. Each of the approximately one dozen compute nodes is configured with approximately 100 processor cores and 1TB of ram. Each job scheduled by LSF runs in a newly allocated Process Authentication Group (PAG). Some jobs run for hours or days; others complete in a few seconds. LSF has been observed scheduling more than 3000 jobs in under a minute with peak counts of cache manager user objects at 1300. Everything ran well until it didn't. After some unexplained trigger event, all of the remaining free memory on the system would be consumed in about an hour causing the system to panic.

Over the last five months AuriStor has diligently worked to develop theories, optimize the AuriStorFS UNIX cache manager to address each theory; then move on to subsequent ideas once prior theories were disproved. Over this time we have replaced the memory management lifecycle for access rights entries and rx events. We have made it possible for Linux permission checks to be processed without acquiring the GLOCK in the majority of access patterns thereby increasing parallelism. The lifecycle management of PAG allocation and user entity tracking has been rewritten to avoid the GLOCK. The Linux inode lifecycle management has been rewritten which some sites attribute to elimination of swapping on their systems. All of these changes increased throughput and parallelism within and without the cache manager. However, none of them prevented the memory consumption once the unknown trigger event occurred.

It is with great pleasure that we can report that the trigger event and the source of the memory consumption have been identified. The trigger is a damaged Rx packet header which causes the AuriStor Rx stack to place an Rx connection into an RX_PROTOCOL_ERROR error state when there is an in-flight call to an OpenAFS fileserver. The memory consumption is due to the infinite loop that occurs when an Rx connection is in an error state and the error code is not one of the error codes which forces replacement of the Rx connection. Within each pass of the loop a structure is allocated to track the history of per-server errors so they can be logged when hard mount mode is enabled for the cell. AuriStor Rx permits up to 12 calls on each Rx connection. As a result there can be up to a dozen processes each spinning and allocating memory when the Rx connection is placed into the error state.

Days before Christmas 2023 AuriStor identified a race condition which can result in damaged Rx DATA packets being written to the network. Initial transmission of Rx DATA packets can be sent either from an application thread or an Rx listener. Packet retransmission occurs either in an Rx listener thread or in the Rx event thread. The race permits a second thread to modify the Rx packet headers while the sendmsg() syscall from the original transmission is executing. This race is possible because the original thread might exhaust its timeslice during the sendmsg() call and it might be many milliseconds before it is rescheduled on a busy system. In the meantime, packet retransmissions might be initiated either by an Rx listener thread processing an ACK packet or the Rx event thread processing a timeout event. If the initial packet transmission completes while the Rx headers are being rewritten the resulting packet sent on the wire might be garbage. AuriStor observed that if the modified packets were jumbograms that silent data corruption could occur so we disabled all use of jumbograms in v2021.05-35 We also notified both IBM and OpenAFS of the problem.

v2021.05-53 modified the RPC result analyze logic to force the replacement of Rx connections in an error state except when the error signals a token expiration. When a token is expired there is no benefit to replacing the connection until new tokens have been acquired. This release further ensures that infinite memory allocation cannot occur by tracking only the most recent error per fileserver; and only does so if hard mount mode is enabled for the cell.

Processes that issued syscalls which became stuck in a RPC retry loop could not be interrupted. In addition to the unintentional retry loops there are intentional retry loops such as those that occur because a fileserver repeated fails the call with a “VBUSY” error. “VBUSY” originally meant that the required volume was temporarily in use by the volume management service (“volserver”) and would be back in service soon so please wait. Beginning with IBM AFS 3.4 the error code was used to force a retry for reasons having nothing to do with the requested volume’s availability. IBM AFS 3.4 introduced the concept of multi-homed clients which meant that the fileserver could no longer track the identity of a cache manager by its IPv4 address. Instead, each cache manager randomly generates a UUID which the fileserver must query each time an incoming RPC is received on a previously unrecognized Rx connection. An Rx connection might be unrecognized because it’s new, or because the client’s IP address changed, or because the fileserver garbage collected its state information. Regardless of the reason, if a call is received and there is no Rx connection binding to a cache manager UUID, then the fileserver application worker thread to which the call was assigned must issue an outgoing RPC to the client’s RXAFSCB service. Each client has at most one worker thread. Prior to this release, every incoming RXAFSCB call required acquisition of the GLOCK before it could execute. If the GLOCK is unavailable, the RXAFSCB thread blocks and no other pending RXAFSCB calls can be processed. If a RXAFSCB call blocks for too long, the fileserver timeouts trigger and the fileserver replies to the incoming RXAFS call with “VBUSY”. “VBUSY” can also be sent if the fileserver’s quota on the number of calls from the same endpoint is exceeded.

The v2021.05-54 release removes the GLOCK requirement for many RXAFSCB RPCs including all of the UUID query RPCs and all whole volume callback RPCs. Callback RPCs for individual vnodes must still acquire the GLOCK. However, it is hoped that these changes will significantly reduce the time required to process the queries generated due to unrecognized Rx connections. The AuriStor Rx stack permits a dozen calls to be issued on a single logical connection which the fileserver sees as three unique Rx connections from the same endpoint.

Each Unix UID or PAG ID must use an independent Rx connection to each fileserver. On a system which processes /afs syscalls for 1200 UID or PAG IDs in a minute, there is the potential for 3600 Rx connections per fileserver. When the fileserver is OpenAFS, this quickly triggers per endpoint throttling and generates VBUSY storms which can result in unlucky client processes experiencing starvation. With this release, starving processes can be interrupted and RXAFSCB UUID queries can be completed without blocking.

AuriStor believes this release will provide significant benefits for large scale compute nodes, web servers and time sharing systems.

v2021.05-53 (4 January 2025)

After executing a call to a fileserver, if the Rx connection is in an error state for any reason other than expired tokens, then schedule the Rx connection to be replaced the next time a call is attempted. All calls issued on an Rx connection that is already in an error state fail immediately preventing further communication with the associated fileserver from the UID/PAG.
Rx connections in an expired token error state are not replaced until new tokens are set.
Rx connections can be placed into an error state either due to local or remote failures. Such failures are often generated by the in-use security class (RXKAD/RXGK) but might be due to an out-of-memory condition, system clock advancement or reversal, or even application errors.
This release includes internal code changes to the management of servers, connections and access rights which will be used by a future release to separate PAG ID allocation from the system's UID namespace.

v2021.05-52 (5 December 2024)

The v2021.05-50 replacement of the linked list implementation of the vcache access cache with a btree and last lookup cache introduced a memory corruption bug which is fixed in this release.
Eliminate the GLOCK requirement for the collection of function call and RPC statistics collection.
Replace small space allocation with a GLOCK free allocator.
Reimplement the cache manager tracing logging routines so they no longer require the GLOCK.
The Rx networking stack schedules many different types of time sensitive events associated with individual packet retransmission, delayed ack and abort processing, path mtu detection, keep alives, and other categories. Prior to this release, all memory allocated to Rx events objects would not be freed to the operating system until the userspace process terminated or the kernel module was unloaded.
Beginning with this release, instead of having a single, global, free list for rxevents, the system allocator is trusted to be able to allocate memory quickly. This simplifies the Rx implementation, and should be more efficient, as most system allocators will have per thread or core slabs, removing the contention of the global lock protecting the rxevent free list. In addition, the change protects against a network event allocating memory that cannot be readily freed.
On non-Total Store Ordering (TSO)architectures, relaxed atomics used as reference counters do not ensure completion of all prior stores or recency of later loads. This release replaces all read-modify-write atomics to be acquire-release, which, among other things, ensures that only the intended ordering is possible.
Alter mutex initialization to permit lockdep debugging (when enabled in the kernel config) to distinguish individual mutexes. Previously lockdep believed yfs.ko only used a single mutex. Additionally, no longer acquire and release mutexes in a helper routine so that lockdep can report warnings within the function that triggered the acquisition.
Process Authentication Group (PAG) IDs are allocated from a 24-bit namespace. To ensure that a PAG ID is not reused more often than once every 200 days, the number of PAGs that are permitted to be allocated are restricted to the number of seconds since the cache manager kernel module was loaded. If a PAG allocation request exceeds the limit, the requesting task blocks in the kernel until the PAG can be allocated. This is referred to as PAG Throttling.
Prior to this release, a task could not be killed or otherwise signaled when blocked in kernel waiting for a PAG ID.
PAG ID allocation is sensitive to clock rollbacks. The logic to detect clock rollbacks has been broken on platforms with 32-bit time_t since it was introduced. This release corrects the rollback detection.

v2021.05-51 (26 November 2024)

Fix a periodic Linux kernel panic that occurs when stopping the AuriStorFS Cache Manager kernel module. Introduced in v2021.05-45.
Reorder the shutdown procedure to ensure that kernel keyring operations cannot be performed against the cache manager module during shutdown.
Prevent a potential deadlock between the cache manager global lock and the VFS. Introduced in OpenAFS 1.4.0.
v2021.05-45 converted the cache manager flags to atomics. The readdir implementation was unintentionally modified to ignore errors returned from the AFS3 directory iterator (with the exception of EINTR and ERESTARTSYS). This can result in use of an uninitialized stack variable. This release restores the pre-v2021.05-45 behavior.
During cache manager module shutdown reset global structures to the module initialization state. Doing so improves the likelihood that the module can be reinitialized.
Fix a vcache (aka inode) reference leak that can occur during an access rights query for a non-directory object if the neither the calling process nor the cache manager possesses tokens for the cell containing the object. The leak was introduced in v2021.05-45.
When scanning the token list for each user searching for expired tokens the user spinlock must be held to prevent races. The updated user management implementation pulled into v2021.05-45 failed to hold the spinlock.

v2021.05-50 (23 November 2024)

Fix a bug introduced in v2021.05-45 which resulted in each access to a volume root directory requiring a full lookup operation.
The linked list implementation of the vcache access rights cache was replaced with a btree and a last lookup cache. The btree design is more efficient for lookups and removals, and scales better as the number of uids and/or process authentication groups (pags) accessing /afs increases.
Prune access rights for expired users. Doing so reduces the cost of subsequent lookups.
Cache the access rights obtained for a uid|pag via the "anyAccess" field. Previously, a uid|pag that obtained its rights from the "anyAccess" field had to perform and fail a rights lookup on each and every vcache access.
When setting or clearing tokens for a single cell, no longer invalidate the access rights cached for the uid|pag for all cells. Instead, only invalidate the access rights cached for the cell whose tokens were set or cleared.

v2021.05-49 (16 November 2024)

The output of "tokens" command failed to report yfs-rxgk tokens was broken starting in v2021.05-46

v2021.05-48 (12 November 2024)

Theft of credentials in Unix client PAGs (CVE-2024-10394)
Cache managers on UNIX platforms where Process Authentication Groups (PAGs) are in use could be at risk of an attacker joining a PAG assigned to another user or service. With control of the PAG, the attacker might retrieve and use credentials representing the identity of another user, or replace the credentials used by the other processes sharing the PAG with credentials of the attacker's choosing.
AuriStorFS cache managers are at lower risk than OpenAFS because AuriStorFS removed the ability to set the PAG of a parent process via 'aklog'; and because AuriStorFS does not support the NFS Translator on Solaris which permitted PAGs to be created by anonymous remote users.
The AuriStorFS v2021.05-48 release removes the last remaining remnants of the cache manager's ability to set the PAG of a parent process.
Fileserver crash and possible information leak on StoreACL/FetchACL (CVE-2024-10396)
A failure to validate AFS3-style ACL strings received over the network impacts fileservers and client utilities with denial of service and potential information disclosure from uninitialized memory access. Vulnerable RPCs include RXAFS_StoreACL, RXAFS_StoreACL2, RXYFS_StoreACL, RXAFS_FetchACL, and RXYFS_FetchACL. These RPCs convey ACLs as a NUL terminated string with TAB and LF characters used as field separators. A malicious authenticated user can submit malformed ACL strings to the fileserver. A malicious administrator could prepare a fileserver to send malformed ACL strings to the clients.
The AuriStorFS RXYFS_StoreOpaqueACL and RXYFS_OpaqueACL RPCs preferred by AuriStorFS cache managers do not convey ACLs as strings and are therefore not vulnerable to abuse. Neither AuriStorFS fileservers nor cache managers communicating with each other are at risk. Mixed OpenAFS and AuriStorFS cache managers and fileservers are at risk.
AuriStorFS fileservers unlike OpenAFS are not at risk of writing malformed ACL strings to the audit log or of leaking memory. However, they were at risk of reading beyond allocated memory which could crash the fileserver or client tools that parse ACL strings.
The AuriStorFS v2021.05-48 performs validity checks on the output of StoreACL and FetchACL RPCs to ensure that the received RPC data is a NUL terminated string, that the NUL terminated string length matches the RPC data length, and strictly enforces the field separator rules.
Preallocated buffer overflows in XDR responses (CVE-2024-10397)
The AuriStorFS and AFS3 RPC suites rely upon Sun RPC XDR to marshal binary data structures for network transfer. The AuriStor XDR implementation is derived from Sun Microsystems' Sun RPC code base. The Sun RPC XDR API permits memory for output parameters to (optionally) be preallocated which can result in various classes of memory corruption and/or memory leaks in RPC initiator processes.
The AuriStorFS v2021.05-48 release introduces additional data length validation checks within the AuriStor XDR implementation and prohibits the use of preallocated memory for string output parameters or fields. All cache managers, servers and command line tools are modified by these changes.

v2021.05-47 (1 November 2024)

Cache Manager:
- The silly-rename logic for clobbering renames introduced in v2021.05-46 failed to initialize filesystem private dentry data on kernels older than 4.5. This failure leads to a kernel panic. This is corrected in this release.
- Force immediate deallocation of evicted inodes. Prior to this change evicted inodes would deallocated or recycled only after the kernel experiences memory pressure.

v2021.05-46 (28 October 2024)

Cache Manager:
- This release includes a major rewrite of the mountpoint evaluation and Linux vfs dentry revalidation logic. POSIX requires that a symlink target string cannot change unless the symlink inode is deleted and replaced with a new symlink inode. Mountpoints are a special type of symlink. In the past, there was an incorrect assumption that if a mountpoint was not replaced that the binding between the dentry and the target volume root directory inode could not have changed.
  The target inode of a mountpoint is determined by the evaluation of the target string in the context of the latest volume location information. Does an location entry exist for the volume name? If so, have the volume IDs changed? Are there valid readonly sites? Is there a backup volume? Any change to the volume location entry might alter the target inode and therefore when the Linux vfs dentry revalidation is requested for a mountpoint it is not safe to skip mountpoint evaluation except when it is is known that none of the mountpoint evaluation inputs have changed.
  Beginning with this release the Linux cache manager is responsive to mountpoint evaluation input changes including volume location entry changes. This is true for both the tradition /afs file namespace and the /afs/.@mount/// (aka magic mount) namespace.
- This release includes a major rewrite of silly-rename processing.
  - Silly rename files could be left behind when fakestat is active; as it is by default.
  - In prior releases, when a rename operation clobbers a directory entry which refers to an in-use inode, a silly rename was not performed. This could result in data loss if the application continues to read from or write to the anonymous inode; or if it creates a new directory entry for the anonymous inode. Without a silly rename the fileserver permanently deletes the vnode when clobbering a directory entry drops the link count to zero.
    Starting with this release the Linux cache manager creates silly rename directory entries when an in-use inode's directory entry is clobbered.
- Previous releases required that the GLOCK be held for all permission rights checks. This release introduces a GLOCK free fastpass for permission rights checks for directory vnodes which have a valid callback.
- Fix a leak of kernel memory while processing setpag syscalls.
- Prevent a kernel memory leak when server preferences are set via the yfs-client.conf [afsd] configuration or via "fs setserverprefs".
- Directory enumeration of a truncated directory now returns an error instead of assuming the end of the directory has been reached.
- Since AFS 3.0, the Unix cache manager has used the root identity credentials to create anonymous outgoing connections to the location service and each fileserver. However, if uid 0 is assigned a token, then those Rx connections will no longer be anonymous. Beginning with this release anonymous outgoing connections are always created with the NOPAG identity (uid 0xffffffff) instead of the root identity.
- When establishing an outgoing rxgk connection, do not fallback to the systemuser's credentials if the user's credentials resulted in a fatal error. Falling back to the systemuser's credentials can result in inappropriate use of an anonymous connection.
- Improved access rights cache correctness for YFS servers
  In prior releases, the access check logic used the file rights for any files fetched from an AuriStorFS fileserver. For files fetched from an AFS-3 fileserver (and, historically, for all files), it used the directory rights, with the (a)dmin right from the file mixed in. The (a)dmin right on a non-directory indicates that the object is owned by the authenticated user.
  This approach has some issues when combined with the access rights cache, and current fileserver callback behaviour. On an AuriStorFS file server, the rights on a non-directory may be determined by the rights granted on its parent directory or, with per-file ACLs, those granted on the object itself. The fileserver will only break a non-directory's callback when a per-file ACL is changed - changing a directory ACL will not break callbacks on files within that directory. This means that changing a directory ACL will not invalidate access rights cache entries on files in tha directory, even if the effective ACL on this files has changed, and the cached rights are no longer correct.
  This release works around this by adding a new function which returns the access rights for a file hosted on an AuriStor fileserver. It uses the parent vnode information to locate the parent directory. If the parent directory isn't in the cache, or it doesn't have a valid callback, or if it has been changed since the file's access rights were cached, it clears the current access rights. Files without a parent directory must have per-file ACLs, and so their cached rights can be safely used.
  Note that files with parent vnodes may still have per-file ACLs, and that the breadcrumbing performed by the client may add parent vnode fields to vnodes which don't have them provided by the fileserver. Such vnodes may have their cached access rights cleared more frequently than necessary.
- Add a new mechanism for caching access rights within the vcache structure. This cache is protected via a vcache specific spinlock, and can be accessed without holding the GLOCK.
  This new cache mechanism returns the memory associated with cached rights back to the kernel's slab free memory pool instead of adding the unused rights structures to a cache manager managed free list. The previous cache implementation never returned allocated memory to the kernel. Instead, invalidated access rights were appended to a free access rights queue for later reuse.
- When a volume is accessed via multiple mountpoints a choice must be made regarding which mountpoint is considered to be the active (or parent) mountpoint. This release alters the behavior such that the active mountpoint is set every time a mountpoint is traversed.
  This behavior is easier to understand and is more likely to provide the expected result for a single process that repeatedly accesses volumes from multiple mountpoints. However, it can result in unexpected results when multiple processes are traversing multiple mountpoints in parallel without any synchronization.

v2021.05-45 (Not released)

v2021.05-44 (17 August 2024)

Cache Manager:
- Since v0.192 the cache manager has failed acquire the global lock when upgrading a shared-lock to a write-lock during the execution of a background cache chunk file truncation.
- Reverted: v2021.05-41 included "For the first time the cache manager can detect the deletion of a volume and handle the creation of a new volume with the same name but a different volume id."
  These changes broke the reliable use of "bind mounts" on Linux. The functionality will be restored in a future release after the "bind mount" failures can be prevented.
- When the disk cache is close to full and the underlying filesystem is slow to truncate files (e.g. xfs) the cache manager may be forced to wait when freeing a discarded disk cache object. If a signal is delivered to the process executing the active syscall during this period, the system might panic after logging the following message:
  yfs: Error freeing discarded dcache
  This release prevents the panic and properly passes the signal to the userspace process.
- During the processing of a syscall by the cache manager kernel module it is often necessary to open, read, write or truncate a disk cache file. Prior to this release if these operations fail the cache manager would panic the system. In the past attempts have been made to ensure that the disk cache can be reliably accessed at run-time by caching 'root' credentials during cache manager initialization. However, on recent Ubuntu distributions there have been reports of kernel panics triggered when operations performed against disk cache files have been blocked by AppArmor even when 'root' credentials are in use.
  As of this release, access to the disk cache will no longer result in a system panic. Instead the active syscall will be denied. Research into why AppArmor is denying access to the disk cache and under what circumstances is on-going.
- arch=x86_64: Modify the AuriStorFS crypto functions written in assembly to permit the Linux objtool to successfully process the compiled functions. If objtool fails to process the functions they are vulnerable to side channel attacks.
Authentication:
- Neither MIT nor Heimdal gssapi nor their gss mechanisms consistently initialize the output 'minorStatus' parameter. Various functions can return either success or failure majorStatus values with minorStatus unassigned. As a result, stack garbage will be used when generating error messages. From now on libyfs_acquire will always initialize the minorStatus output variable to zero before calling into the gssapi library.
Command Parser:
- No longer accept the token "-" as a switch which eventually fails with a CMD_UNKNOWNSWITCH error. Instead, process the token as a data value.
- Optimize the processing of the loop which processes "source" command input.
- If the source command input file is "-", read from stdin.

v2021.05-41 (26 June 2024)

Linux 6.10 kernels support.
Rx Networking (libyfs_rx):
- A race during event creation can lead to the freeing of the event while its still in use.
- RFC1122 says that Net and Host unreachable ICMP errors might be transient and should therefore not be treated as fatal. There is no such language for the equivalent ICMPV6 errors. However, in practice ICMP6_DST_UNREACH_NOROUTE, ICMP6_DST_UNREACH_BEYONDSCOPE, and ICMP6_DST_UNREACH_ADDR can be transient.
  Linux has considered these ICMPV6 destination unreachable errors as non-fatal going back at least as far as the initial git repository commit.
  AuriStor Rx has always treated these as fatal errors which results in immediate termination of in-flight calls when received. Even if the network route corrects itself before the call timeout period expires. This release mirrors the Linux behavior and makes these errors non-fatal.
Cache Manager:
- For the first time the cache manager can detect the deletion of a volume and handle the creation of a new volume with the same name but a different volume id.
- If the location service reports the deletion of a volume, invalidate all mount points to that volume.
- RXAFS_GetCapabilities RPC failures should not be treated as a fatal error preventing failover to another replica site.
Authentication ("libyfs_acquire") used by aklog, vos, pts, bos, afsio:
- rxkad_k5 token acquisition krb5 ccache management. This release altered the krb5 credential cache management strategy once again to work around different bugs in MIT krb5 and Heimdal.
- New ACQUIRE_ERR_CRED_EXPIRED error code introduced to represent the case when a request for a service credential returns one that is already expired.
Command parser (libyfs_cmd):
- When parsing configuration files there is a depth limit of ten active inclusions. This limit was improperly enforced as a limit of ten included files instead of a depth of ten included files. As of this release it is now possible to populate an includedir directory with any number of .conf files.

v2021.05-40

Not released.

v2021.05-39 (20 May 2024)

Linux 6.9 kernels support.
Parallel Random Number Generation:
- AuriStorFS processes rely upon the krb5_generate_random() and RAND_bytes() functions to obtain random bytes for cryptographic operations and random counters. krb5_generate_random() internally acquires a mutex to protect internal state information. This mutex has become a significant barrier to the encryption and checksumming of Rx packets with both yfs-rxgk and rxkad.
  
  This release replaces general use of krb5_generate_random() and RAND_bytes() with a per-thread ChaCha20 CS-PRNG. This avoids the acquisition of a global mutex and permits increased parallelism on multi-core systems.
Rx Networking (libyfs_rx):
- The Rx network stack schedules a garbage collection operation to execute once per minute. This operation enforces call timeouts, destroys idle connections and destroys idle peers. The operation has historically been performed by the Rx event thread which is already responsible for performing actions in response to call RTOs, sending NAT Ping and keep-alive packets, and retrying connection challenge and reachability checks.
  
  The time complexity of the garbage collection operation is determined by the number of calls, connections, and peers. The busier the Rx endpoint the more work must be performed during each garbage collection run and the longer it takes to complete. While garbage collection is active other events cannot be processed which can interfere with the proper flow control of active calls.
  
  As with all Rx events, the garbage collection event is scheduled to execute at an absolute clock time. If the system clock drifts (or is administratively set) backwards garbage collection will not be performed until the clock catches up with the scheduled time.
  
  Another responsibility of the garbage collection procedure is to terminate calls if the system clock drifted backwards by five minutes or longer. However, when the clocked drifts backwards garbage collection is not performed until the clock has advanced beyond the point where calls require termination. As a result, calls are not terminated due to backwards clock drift and they can stall.
  
  This release re-implements the garbage collection procedure using a dedicated thread and relative waits. This change ensures that the garbage collection procedure will not prevent the execution of call related events and permits calls to be terminated when large backward clock drifts are detected.
Disk Cache Management:
- Since IBM AFS 3.5, the cache has been considered "too full" even if there exist cache files have been discarded but not yet truncated. When the cache is "too full" most operations that write to the cache will block until truncation of discarded cache files has been performed which results in unnecessary delays. This release fixes the cache such that that discarded but not yet truncated cache files do not block write operations.
- This release permits the cache truncation daemon thread to exit sooner if the cache manager is shutting down.
Fixes for two potential kernel bugs.
Improved failover when the RXGK service (co-located with each vlserver) fails to issue tokens. The failures might be the result of misconfiguration, an inability to read keys or loss of Ubik quorum.
Authentication ("libyfs_acquire") used by aklog, vos, pts, bos, afsio:
- Depending upon the Kerberos v5 credential cache implementation it is possible that more than one afs/CELL@REALM service ticket will be added to the credential cache for each execution of aklog or acquisition of rxkad tokens by bos, pts, vos and other administration tools. As the number of service tickets within a cache increases the cost of finding a matching service ticket increases.
  
  The rxkad token acquisition logic explicitly requested that Kerberos v5 afs/CELL@REALM service tickets have a lifetime not to exceed 30-days. If a service ticket lifetime is longer than 30-days it will be rejected by the contacted service without any ability to log the reason for the failure. Unfortunately, by explicitly requesting a maximum endtime, the MIT krb5 implementation ignores any valid matching afs/CELL@REALM service ticket unless there is an exact match, and a new request will be issued to the KDC.
  
  This release alters the logic to request a service ticket without any explicit maximum lifetime. After the service ticket is obtained a maximum lifetime validation check is performed. If the lifetime exceeds 30-days, then
  - an attempt is made to delete the ticket from the credential cache but not all credential cache implementations support cache entry deletion.
  - a new service ticket with an explicit endtime is requested but the previously obtained service ticket might be returned from the cache.
  - the 30-day lifetime validation check is performed again and if it fails then an error message is constructed indicating that the KDC service principal maximum lifetime must be restricted to 30-days.
  This change will avoid acquiring new service tickets from the KDC unless there is no existing ticket or the existing ticket has expired.

v2021.05-38 (29 February 2024)

As with other AuriStorFS releases since the beginning of 2024, this release includes additional improvements to the Rx RPC implementation which are related to the possibility of silent data corruption when Rx jumbograms are in use. Prior releases disabled the negotiation of Rx jumbograms such that the v2021.05-37 Rx peer will refuse to send Rx jumbograms and will request that the remote peer does not send them. However, a bad actor could choose to send Rx jumbograms even though they were asked not to. v2021.05-38 introduces additional protections to ensure that a corrupt Rx jumbogram is dropped instead of being accepted.

The v2021.05-38 Rx RPC implementation also includes two optimizations. First, when Rx initiators complete a call they will no longer send an extra ACK packet to the Rx acceptor of the completed call. The sending of this unnecessary ACK creates additional work for the server which can result in increased latency for other calls being processed by the server.

Second, all AuriStor Rx services require a reach check for incoming calls from Rx peers to help protect against Distributed Reflection Denial of Service (DRDoS) attacks and execution of RPCs when the response cannot be delivered to the caller. A new reach check is required for each new call that arrives more than 60 seconds after the prior reach check completed. v2021.05-38 Rx considers the successful acknowledgment of a response DATA packet as a reach check validation. With this change reach checks will not be periodically required for a peer that completes at least one call per 60 seconds. A 1 RTT delay is therefore avoided each time a reach check can be avoided. In addition, reach checks require the service to process an additional ACK packet. Eliminating a large number of reach checks can improve overall service performance.

The final Rx RPC change in this release is specific to kernel implementations. Prior releases restricted the frequency of executing time scheduled Rx events to a granularity no smaller than 500ms. As a result an RTO timer event for a lost packet could not be shorter than 500ms even if the measured RTT for the connection is significantly smaller. The minimum RTO for a connection in AuriStor Rx is 200ms. The inability to schedule shorter timeouts impacts recovery from packet loss.

For client systems, the v2021.05-38 release contains fixes for two bugs that have resulted in system crashes on Linux when resource limits have been exceeded either by the system as a whole or for the process accessing /afs.

CrayOS SLES 5.14.21 is now a supported client platform.

v2021.05-37 (5 February 2024)

Linux 6.8 kernels support.
Rx improvements:
- The v2021.05-36 release permanently disabled all use of Rx jumbograms due to a risk of silent data corruption. However, when advertising the number of acceptable datagrams in the ACK trailer a missing htonl() set the value to 16777216 instead of 1 on little-endian systems.
- When sending a PING ACK as a reachability test, ensure that the previousPacket field is properly assigned to the largest accepted DATA packet sequence number instead of zero.
- Replace the initialization state flag with two flags. One that indicates that Rx initialization began and the other that it succeeded. The first prevents multiple attempts at initialization after failure. The second prevents shutdown from accessing uninitialized structures if initialization failed.
Cache Manager Improvements:
- No longer refuse to start if both the 'cachedir' and 'memcache' options are present in the configuration file.
- New variable to store the maximum number of cache blocks used. which is accessible via /proc/fs/auristorfs/cache/blocks_used_max.

v2021.05-36 (10 January 2024)

Rx improvements:
- Permanently disable all use of Rx jumbograms due to a risk of silent data corruption.
  
  Ever since OpenAFS 1.0, and possibly before, a race condition has existed when Rx transmits packets. As the rx_call.lock is dropped when starting packet transmission, there is no protection for data that is being copied into the kernel by sendmsg(). It is critical that this packet data is not modified by another thread. However, races exist between the application, listener, and event threads that can lead to retransmissions starting whilst an original transmission is still in progress. This can lead to the packet headers being overwritten, and either the original transmission, the retransmission or both sending corrupt data to the peer.
  
  This corruption can affect the packet serial number or packet flags. It is particularly harmful when the packet flags are corrupted, as this can lead to multiple Rx packets which were intended to be sent as Rx jumbograms being delivered and misinterpreted as a single large packet. The eventual result of this depends on the Rx security class in play, but it can cause decrypt integrity errors (rxgk:crypt and rxgk:auth) or corruption of the data stream (rxnull, rxgk:clear or rxkad:auth).
  
  All AuriStorFS servers, OpenAFS 1.6 or later servers, and the Windows cache manager have been shipped with Rx jumbograms disabled by default. The UNIX cache managers however are shipped with jumbograms enabled. There are many AFS cells around the world that continue to deploy OpenAFS 1.4 or earlier fileservers which continue to negotiate the use of Rx jumbograms.
  
  It is worth noting that all AuriStorFS v0.198 and later fileservers and cache managers implement explicit checks that will recognize the corrupted application data stream and prevent corrupted file content from being stored either into an AFS vnode's backing store in the volume's object store or the cache manager's AFS cache. OpenAFS cache managers and fileservers do not have these protections.
  
  With Rx jumbograms disabled the maximum number of Rx packets in a datagram is reduced from 6 to 1; the maximum number of send and receive datagram fragments is reduced from 4 to 1; and the maximum advertised MTU is restricted to 1444 - the maximum rx packet size prior to the introduction of jumbograms in IBM AFS 3.5.
- If the rx call flow state transitions from either the RECOVERY or RESCUE states to the LOSS state as a result of an RTO resend event while writing packets to the network, cease transmission of any new DATA packets if there are packets in the resend queue.
- When the call flow state is LOSS and all packets in the resend queue have been retransmitted and yet the recovery point has not been reached, then permit new DATA packets to be sent in order to maintain a full congestion window.
- Add a safety check to prevent the estimated RTT from underflowing when the actual roundtrip time is smaller than 125us.
- Fix the computation of the padding required for rxgk encrypted packets. This bug results in packets sending 8 bytes fewer per packets than the network permits. This bug accidentally prevented the construction of Rx jumbograms when a call is protected by rxgk:crypt.
- Replace the random number generator with a more security source of random bytes.
Cache Manager Improvements:
- In Linux kernels with folio mapping functionality, prior releases of AuriStorFS cache manager could trigger an infinite loop when getting a page. This release converts to using the new folio mapping functionality instead of page mapping when available.
- afsd will now log the set of network interfaces in use whether or not rxbind is configured.
- afsd will no longer drop user-defined mount options if SELinux is disabled.
- Prevent possible memory corruption when listing tokens.

New v2021.05-34 (21 December 2023)

Cache Manager:
- v2021.05-33 introduced a critical bug for Linux cache managers. Creating a hard link produces an undercount of the linked inode's i_count. This undercount can result in a kernel module assertion failure if the inode is garbage collected due to memory pressure. The following message will be logged to dmesg
```
     "yfs: inode freed while on LRU"
   
```
  followed by a kernel BUG report. This bug is fixed in v2021.05-34.
- If the oom-killer terminates a process while it is executing within the AuriStorFS kernel module it is possible for memory allocations to fail. This can lead to failures reading from the auristorfs cache. This release includes additional logic to permit failing the cache request without triggering a NULL pointer dereference.
- If the auristorfs disk cache filesystem is remounted read-only then the disk cache will become unusable. Instead of triggering a system panic when attempts to read or write fail, log a warning and fail the request.

New v2021.05-33 (27 November 2023)

Cache Manager:
- Linux 6.7 kernel support
- When creating a hard link, the new directory entry must refer to the target inode in order for the dentry to be "positive". Previously this linkage was delayed until a subsequent revalidation of the dentry.
- Always use the file_dentry() helper to evaluate the target dentry When overlayfs is in use, the failure to use file_dentry() can result in use of the wrong dentry.
- Restrict the use of the d_automount mechanism to volume root directory inodes. The d_automount mechanism does not apply to non-root directories and can interfere with use of AuriStorFS volumes and overlayfs.
- Restore rename flag validation for kernels that support mnt_idmap.
Rx improvements:
- Not all calls transfer enough data to be able to measure a smoothed round-trip time (SRTT). Calls which are unable to compute a SRTT should not be used to update the peer host RTO value which is used to initialize the RTO for subsequent calls.
  
  Without this change, a single DATA packet call will cause the peer host RTO to be reduced to 0ms. Subsequent calls will start with a RTO value of MAX(0, rxi_minPeerTimeout) where rxi_minPeerTimeout defaults to 200ms. If the actual measured RTO is greater than 200ms, then initial RTO will be too small resulting in premature triggering of the RTO timer and the call flow state entering the loss phase which can significantly hurt performance.
- Initialize the peer host RTO to rxi_minPeerTimeout (which defaults to 200ms) instead of one second. Although RFC6298 recommends the use of one second when no SRTT is available, Rx has long used the rxi_minPeerTimeout value for other purposes which are supposed to be consistent with initial RTO value. It should be noted that Linux TCP uses 200ms instead of one second for this purpose.
- If associating a security class with an Rx connection fails immediately place the Rx connection into an error state. A failure might occur if the security class is unable to access valid key material.
- If an incoming Rx call requires authentication and the security class is unable to successfully generate a challenge, put the incoming Rx connection into an error state and issue an abort to the caller.
- If an incoming Rx call requires authentication and the security class is able to generate a challenge but the challenge cannot be returned to Rx, then treat this as a transient error. Do not acknowledge the incoming DATA packet and do not place the Rx connection into an error state. An attempt to re-issue the challenge will be performed when the DATA packet is retransmitted.
- If an Rx call is terminated due to the expiration of the configured connection dead time, idle dead time, hard dead time, or as a result of clock drift, then send an ABORT to the peer notifying them that the call has been terminated. This is particularly important for terminated outgoing calls. If the peer does not know to terminate the call, then the call channel might be in use when the next outgoing call is issued using the same call channel. If the next incoming call is received by an in-use call channel, the receiver must drop the received DATA packet and return a BUSY packet. The call initiator will need to wait for a retransmission timeout to pass before retransmitting the DATA packet. Receipt of BUSY packets cannot be used to keep a call alive and therefore the requested call is at greater risk of timing out if the network path is congested.
aklog and krb5.log (via libyfs_acquire):
- If the linked Kerberos library implements krb5_cc_cache_match() and libacquire has been told to use an explicit principal name and credential cache, the Kerberos library might return KRB5_CC_NOTFOUND even though the requested credential cache is the correct one to use. This release will not call krb5_cc_cache_match() if the requested credential cache contains the requested principal.
Cell Service Database (cellservdb.conf):
- cellservdb.conf has been synchronized with the 31 Oct 2023 update to the grand.central.org CellServDB file.

v2021.05-32 (9 October 2023)

Cache Manager:
- CRITICAL UPDATE for aarch64 systems. Prior releases incorrectly compiled Neon source code routines and as a result floating point errors can occur.
- The d_revalidate dentry operation should return false if the fileserver reports a FileID as non-existent in response to an InlineBulkStatus or FetchStatus RPC.
aklog and klog.krb5:
- Only output an error message if the token cannot be set into neither the AuriStorFS cache manager nor the Linux kernel afs cache manager.

v2021.05-31 (25 September 2023)

New platform:
- Linux 6.6 kernels
Cache Manager:
- If an AuriStorFS cache manager is unable to use the yfs-rxgk security class when communicating with an AuriStorFS fileserver, it must assume it is IBM AFS 3.6 or OpenAFS and upgrade it to AuriStorFS if an upgrade probe returns a positive result. Once a fileserver's type is identified as AuriStorFS the type should never be reset; even if communication with the fileserver is lost or the fileserver restarts.
  
  If an AuriStorFS fileserver is replaced by an OpenAFS fileserver on the same endpoint, then the UUID of the OpenAFS must be different. As a result, the OpenAFS fileserver will be observed as distinct from the AuriStorFS fileserver that previously shared the endpoint.
  
  Prior to this release there were circumstances in which the cache manager discarded the fileserver type information and would fail to recognize the fileserver as an AuriStorFS fileserver when yfs-rxgk could not be used. This release prevents the cache manager from resetting the type information if the fileserver is marked down.
- If a fileserver's location service entry is updated with a new uniquifier value (aka version number), this indicates that one of the following might have changed:
  1. the fileserver's capabilities
  2. the fileserver's security policy
  3. the fileserver's knowledge of the cell-wide yfs-rxgk key
  4. the fileserver's endpoints
  Beginning with this release the cache manager will force the establishment of new Rx connections to the fileserver when the uniquifier changes. This ensures that the cache manager will attempt to fetch new per-fileserver yfs-rxgk tokens from the cell's RXGK service, enforce the latest security policy, and not end up in a situation where its existing tokens cannot be used to communicate with the fileserver.
- Red Hat EL9 kmods are now bound to the explicit kernel version they were built for. EL9 kmods for prior releases failed to include the appropriate bindings.
- Do not use a weak ref to key_type_keyring on aarch64. Doing so can result in a failure to load the module.
  
  module yfs: unsupported RELA relocation: 311
aklog:
- Fix incorrect output when populating the server list for a service fails. The stashed extended error explaining the cause of the failure was not displayed.
- If a cell has neither _afs3-prserver._udp. DNS SRV records nor AFSDB records, the lookup of the cell's protection servers would fail if there is no local cell configuration details. The fallback to use _afs3-vlserver._udp. DNS SRV records did not work. This is corrected in this release.
- When setting tokens for use by the Linux kafs kernel module, the syscall error handling was broken which could result in a report of successful token insertion when in fact the syscall failed.

v2021.05-30 (6 September 2023)

New platform: Linux 6.5 kernels
Do not mark a fileserver down in response to a KRB5 error code.
fs cleanacl must not store back to the file server a cleaned acl if it was inherited from a directory. Doing so will create a file acl.
Correct the generation of never expire rxkad_krb5 tokens from Kerberos v5 tickets which must have a start time of Unix epoch and an end time of 0xFFFFFFFF seconds. The incorrectly generated tokens were subject to the maximum lifetime of 30 days.
Correct the generation of the yfs-rxgk RESPONSE packet header which failed to specify the key version generation number used to encrypt the authenticator. If the actual key version is greater than zero, then the authenticator would fail to verify.
Enforce a maximum NAT ping period of 20s to ensure that NAT/PAT/firewall rules due not expire while Rx RPCs are in-flight.

v2021.05-29 (26 June 2023)

New platform: Linux 6.4 kernels
Execution of fs commands such as examine, whereis, listquota, fetchacl, cleanacl, storeacl, whoami, lsmount, bypassthreshold and getserverprefs could result in memory leaks by the yfs.ko kernel module
Prevent a kernel panic if the configured cache directory is located on a filesystem such as overlayfs which does not support the functionality required to be a cache

v2021.05-28 (10 May 2023)

No changes compared to v2021.05-27.

v2021.05-27 (1 May 2023)

Fixes for bugs in vos introduced in v2021.05-26.

v2021.05-26 (17 April 2023)

New Platform: Linux mainline kernels 6.3
New Platform: Red Hat Fedora 38
New Platform: Red Hat EL8 Real Time kernels
New Platform: Red Hat EL9 Real Time kernels
New Repository: Red Hat EL8 aarch64
New Repository: Red Hat EL9 aarch64
New Repository: Red Hat Fedora 38 aarch64
Fixed a potential kernel memory leak when triggered by fs examine, fs listquota, or fs quota.
Increased logging of VBUSY, VOFFLINE, VSALVAGE, and RX_RESTARTING error responses. A log message is now generated whenever a task begins to wait as a result of one of these error responses from a fileserver. Previously, a message was only logged if the volume location information was expired or discarded.
Several changes to optimize internal volume lookups.
Faster failover to replica sites when a fileserver returns RX_RESTARTING, VNOVOL or VMOVED.
rxdebug regains the ability to report rx call flags and rx_connection flags.
The RXRPC library now terminates calls in the QUEUED state when an ABORT packet is received. This clears the call channel making it available to accept another call and reduces the work load on the worker thread pool.
Fileserver endpoint registration changes no longer result in local invalidation of callbacks from that server.
Receipt of an RXAFSCB_InitCallBackState3 RPC from a fileserver no longer resets the volume site status information for all volumes on all servers.
If SELinux is disabled, afsd will disable the setting of mount options for SELinux contexts. Since the introduction of SELinux mount options, the AuriStorFS cache manager could not be started if SELinux was disabled.
New sysctl variables rx_harddeadtime, rx_idledeadtime, and rx_idledeadtime_replicated which join rx_deadtime. These are read/write variables permitting the rx connection dead time values to be adjusted at run-time.
The [afsd] ignorelist-dns entries are now compared to lookup strings in a case insensitive manner as DNS lookups are case insensitive.
Linux 6.1 and 6.2 kernels could "oops" during suspend operations.
Restore support for arm8-a architecture systems such as AMD Seattle (Rev.B1). Support for arm8-a was unintentially disabled in v2021.05-19.
Truncating a file larger than 4GB to a size larger than 4GB (e.g. from 6GB to 4GB) will result in the file being truncated to a file smaller than 4GB.
fs cleanacl would crash after the acl cleaning was performed.

v2021.05-25 (28 December 2022)

New Platform: Linux mainline kernels 6.1 and 6.2
The v2021.05-25 release includes further changes to RXRPC to improve reliability. The changes in this release prevent improper packet size growth. Packet size growth should never occur when a call is attempting to recover from packet loss; and is unsafe when the network path's maximum transmission unit is unknown. Packet size growth with be re-enabled in a future AuriStorFS release that includes Path MTU detection and the Extended SACK functionality.
With this release the Linux /proc/fs/yfs directory tree has been moved to /proc/fs/auristorfs. A symlink from /proc/fs/yfs to /proc/fs/auristorfs is provided to ensure backward compatibility.
A new /proc/fs/auristorfs/rxstats file can be used to read the RX statistics counters. This set of statistics uses 64-bit counters unlike the output from "rxdebug -rxstat" which is limited to 32-bit counters.
Improved error text describing the source of invalid values in /etc/yfs/yfs-client.conf or included files and directories.

AlmaLinux and Rocky Linux Repositories added (2 November 2022)

v2021.05-23 (4 October 2022)

New Platform: Fedora 37
Linux Kernel Module
- Enable the use of cell aliases when evaluating magic mount paths
RX RPC
- The number of sent ABORT packets have not been counted for a long time. Count the sent ABORT packets and deliver the count in response to an rxdebug server port -rxstats query.
- RX calls are now created with a fixed initial congestion window instead of using a value stashed from a prior call. Use of a stashed value was introduced in IBM AFS 3.5. The stashed value can slow the transfer rate of subsequent calls and is not consistent with RFC5681.

v2021.05-22 (12 September 2022) and v2021.05-21 (6 September 2022)

Linux Kernel Module
- Linux mainline kernel 6.0 is supported
- Fix a build error with 5.19 or later kernels when the architecture is aarch64.
- The cache manager kernel module now includes description, author and version information that can be displayed via modinfo.
RX RPC
- If receipt of a DATA packet causes an RX call to enter an error state, do not send the ACK of the DATA packet following the ABORT packet. Only send the ABORT packet.
- AuriStor RX has failed to count and report the number of RX BUSY packets that have been sent. Beginning with this change the sent RX BUSY packet count is once again included in the statistics retrieved via rxdebug server port -rxstats.
- Introduce minimum and maximum bounds checks on the ACK packet trailer fields. If the advertised values are out of bounds for the receiving RX stack, do not abort the call but adjust the values to be consistent with the local RX RPC implementation limits. These changes are necessary to handle broken RX RPC implementations or prevent manipulation by attackers.

New to v2021.05-20 (15 August 2022) and v2021.05-19 (13 August 2022)

RX RPC
- Include the DATA packet serial number in the transmitted reachability check PING ACK. This permits the reachability test ACK to be used for RTT measurement.
- Do not terminate a call due to an idle dead timeout if there is data pending in the receive queue when the timeout period expires. Instead deliver the received data to the application. This change prevents idle dead timeouts on slow lossy network paths.
- Fix assignment of RX DATA, CHALLENGE, and RESPONSE packet serial numbers in Linux (userspace). Due to a mistake in the implementation of atomic_add_and_read the wrong serial numbers were assigned to outgoing packets.

New to v2021.05-18 (12 July 2022)

Cache Manager
- Linux Kernel 5.19 is now supported
- Prevent a kernel memory leak of less than 64 bytes for each bulkstat RPC issued to a fileserver. Bulkstat RPCs can be frequently issued and over time this small leak can consume a large amount of kernel memory. Leak introduced in AuriStorFS v0.196.
- The Perl::AFS module directly executes pioctls via the OpenAFS compatibility pioctl interface instead of the AuriStorFS pioctl interface. When Perl::AFS is used to store an access control list (ACL), the deprecated RXAFS_StoreACL RPC would be used in place of the newer RXAFS_StoreACL2 or RXYFS_StoreOpaqueACL2 RPCs. This release alters the behavior of the cache manager to use the newer RPCs if available on the fileserver and fallback to the deprecated RPC. The use of the deprecated RPC was restricted to use of the OpenAFS pioctl interface.
RX RPC
- Handle a race during RX connection pool probes that could have resulted in the wrong RX Service ID being returned for a contacted service. Failure to identify that correct service id can result in a degradation of service.
- The Path MTU detection logic sends padded PING ACK packets and requests a PING_RESPONSE ACK be sent if received. This permits the sender of the PING to probe the maximum transmission unit of the path. Under some circumstances attempts were made to send negative padding which resulted in a failure when sending the PING ACK. As a result, the Path MTU could not be measured. This release prevents the use of negative padding.
Some shells append a slash to an expanded directory name in response to tab completion. These trailing slashes interfered with "fs lsmount", "fs flushmount" and "fs removeacl" processing. This release includes a change to prevent these commands from breaking when presented a trailing slash.

New to v2021.05-17 (16 May 2022)

Cell Service Database Updates
- Update cern.ch, ics.muni.cz, ifh.de, cs.cmu.edu, qatar.cmu.edu, it.kth.se
- Remove uni-hohenheim.de, rz-uni-jena.de, mathematik.uni-stuttgart.de, stud.mathematik.uni-stuttgart.de, wam.umd.edu
- Add ee.cooper.edu
- Restore ams.cern.ch, md.kth.se, italia
Fix parsing of [afsd] rxwindow configuration which can be used to specified a non-default send/receive RX window size. The current default is 128 packets.
RX Updates
- Add nPacketsReflected

Logging improvements

Cache directory validation errors log messages now include the cache directory path.

Log the active configuration path if "debug" logging is enabled.

More details of rxgk token extraction failures.

New to v2021.05-16 (24 March 2022)

RX - Previous releases re-armed the Retransmission Timeout (RTO) each time a new unacknowledged packet is acknowledged instead of when a new leading edge packet is acknowledged. If leading edge data packet and its retransmission are lost, the call can remain in the "recovery" state where it continues to send new data packets until one of the following is true:
. the maximum window size is reached
. the number of lost and resent packets equals 'cwind'
at which point there is nothing left to transmit. The leading edge data packet can only be retransmitted when entering the "loss" state but since the RTO is reset with each acknowledged packet the call stalls for one RTO period after the last transmitted data packet is acknowledged.

This poor behavior is less noticiable with small window sizes and short lived calls. However, as window sizes and round-trip times increase the impact of a twice lost packet becomes significant.

RX - Never set the high-order bit of the Connection Epoch field. RX peers starting with IBM AFS 3.1b through AuriStor RX v0.191 ignore the source endpoint when matching incoming packets to RX connections if the high-order epoch bit is set. Ignoring the source endpoint is problematic because it can result in a call entering a zombie state whereby all PING ACK packets are immediately responded to the source endpoint of the PING ACK but any delayed ACK or DATA packets are sent to the endpoint bound to the RX connection. An RX client that moves from one network to another or which has a NAT|PAT device between it and the service can find themselves stuck.

Starting with AuriStor RX v0.192 the high-order bit is ignored by AuriStor RX peer when receiving packets. This change to always clear the bit prevents IBM AFS and OpenAFS peers from ignoring the source endpoint.

RX - The initial packetSize calculation for a call is altered to require that all constructed packets before the receipt of the first ACK packet are eligible for use in jumbograms if and only if the local RX stack has jumbograms enabled and the maximum MTU is large enough. By default jumbograms are disabled for all AuriStorFS services. This change will have a beneficial impact if jumbograms are enabled via configuration; or when testing RX performance with "rxperf".

New fs whereis -noresolve option displays the fileservers by network endpoint instead of DNS PTR record hostname.

aklog - if -cache is not specified, fetch the ccache name from the KRB5CCNAME environment variable if present. If aklog is started in a secure environment (e.g. from sshd pam) libkrb5 will be unable to read the KRB5CCNAME environment variable when selected the default ccache.

New to v2021.05-15 (24 January 2022)

kernel - fixed YFS_RXGK service rx connection pool leak

New to v2021.05-14 (20 January 2022)

Support Linux 5.17 kernels

fs mkmount permit mount point target strings longer than 63 characters.
Linux 5.13 and later kernels: Prevent oops when seeking the afs_ioctl special files. A BUG will be generated to dmesg log but there appear to be no other adverse affects.

kernel - Prevent splice operation recursion which can lead to failed RPCs and data corruption. On 5.13 and later kernels 'cp' is implemented using the copy_file_range syscall which begins a splice operation. As a result it is not safe for the cache manager to use splice when reading from or writing to the afs disk cache.

afsd enhance logging of yfs-rxgk token renewal errors.

afsd gains a "principal = configuration option for use with keytab acquisition of yfs-rxgk tokens for the cache manager identity.

kernel - Avoid unnecessary rx connection replacement by racing threads after token replacement or expiration.

kernel - Fix a regression introduced in v2021.05 where an anonymous combined identity yfs-rxgk token would be replaced after three minutes resulting in the connection switching from yfs-rxgk to rxnull.

kernel - Fix a regression introduced in v0.208 which prevented the invalidation of cached access rights in response to a fileserver callback rpc. The cache would be updated after the first FetchStatus rpc after invalidation.

kernel - Reset combined identity yfs-rxgk tokens when the system token is replaced.

kernel - The replacement of rx connection bundles in the cache manager to permit more than four simultaneous rx calls per uid/pag with trunked rx connections introduced the following regressions in v2021.05.

a memory leak of discarded rx connection objects

failure of NAT ping probes after replacement of an connection

inappropriate use of rx connections after a service upgrade failure

All of these regressions are fixed in patch 14.

kernel - afsd can be started with the nomount option with the intention that a mount command will be performed asynchronously. If the mount is performed before afsd starts it would block. Now it will fail after a 5 second wait.

kernel - add /proc/sys/fs/auristorfs/mountable which reads 0 if the filesystem is not ready to mount and 1 if it is.

New to v2021.05-12 (7 October 2021)

Support Linux 5.16 kernels.

CRITICAL (rhel7): All rhel 7 kernels starting from 3.10.0_861.el7 through 3.10.0_1160.49.1.el7 contain a broken implementation of generic_file_aio_read() which can return 0 bytes even though the end of the file stream has not been reached. This bug can cause a variety of unpredictable failure conditions when satisfying a vfs request requires reading from a disk cache. AuriStorFS v2021.05-10 and later includes a workaround for the broken behavior. Fixed in kernel-3.10.0-1160.51.1.el7.

fs ignorelist -type afsmountdir in prior releases could prevent access to /afs.

Location server rpc timeout restored to two minutes instead of twenty minutes.

Location server reachability probe timeout restored to six seconds instead of fifty seconds.

Cell location server upcall results are now cached for fifteen seconds.

Multiple kernel threads waiting for updated cell location server reachability probes now share the results of a single probe.

RX RPC implementation lock hierarchy modified to prevent a lock inversion.

RX RPC client connection reference count leak fixed.

RX RPC deadlock during failed connection service upgrade attempt fixed..

New to v2021.05-9 (25 October 2021)

Support Linux 5.15 kernels.

Linux cache managers configured with a disk cache store the contents of the dynamic root (dynroot) /afs directory in a disk cache chunk even though the directory is not fetched from a fileserver. Doing so permits the dynroot directory to be iterated using the same directory processing framework as fileserver stored directory objects. After an "afsd" restart if the first vfs access to the dynroot directory occurs when the dynamic data version matches the on-disk directory data version the on-disk data will be treated as if it were current. The on-disk "dynroot" content cannot be trusted after a restart and therefore must be discarded during disk cache initialization.

A corrupted cache manager directory buffer can result in an Unexpected directory iteration error with code EINVAL when attempting to parse the directory buffer contents. The corrupted buffer might remain in memory for an extended period of time if the directory access failure is repeatedly retried in response to syscalls.
Now, a transient failure reading from the directory disk cache chunk will result in the logging of code EIO an the damaged buffer will not be cached as a valid directory buffer.

Update "afsio" to support reading and writing files larger than 2GB.

New to v2021.05-7 (22 August 2021)

Linux 5.14 kernel support

Introduce a disk cache filesystem usability test permit early failure detection in the case of readonly or remote filesystems.

Fix "klog -setpag". A pag might have been created when not requested.

Miscellaneous RX updates.

Prevent theoretical deadlock when evaluating @sys component names.

Improve logging of "afsd" startup before and after daemonization.

Fix "afsio" to correctly write files larger than 64MB.

Fix in kernel credential reference count bug.

New to v2021.05-3 (10 June 2021)

Fix for [cells] cellname = {...} without server list.

New to v2021.05 (31 May 2021)

Multi-homed location servers are finally managed as a single server instead of treating each endpoint as a separate server. The new functionality is a part of the wholesale replacement of the former cell management infrastructure. Location server communication is now entirely managed as a cluster of multi-homed servers for each cell. The new infrastructure does not rely upon the global lock for thread safety.

This release introduces a new infrastructure for managing user/pag entities and tracking their per cell tokens and related connection pools.

Expired tokens are no longer immediately deleted so that its possible for them to be listed by "tokens" for up to two hours.

Prevent a lock inversion introduced in v0.208 that can result in a deadlock involving the GLOCK and the rx call.lock. The deadlock can occur if a cell's list of location servers expires and during the rebuild an rx abort is issued.

Add support for rxkad "auth" mode rx connections in addition to "clear" and "crypt". "auth" mode provides integrity protection without privacy.

Add support for yfs-rxgk "clear" and "auth" rx connection modes.

Do not leak a directory buffer page reference when populating a directory page fails.

Re-initialize state when populating a disk cache entry using the fast path fails and a retry is performed using the slow path. If the data version changes between the attempts it is possible for truncated disk cache data to be treated as valid.

Log warnings if a directory lookup operation fails with an EIO error. An EIO error indicates that an invalid directory header, page header, or directory entry was found.

Do not overwrite RX errors with local errors during Direct-I/O and StoreMini operations. Doing so can result in loss of VBUSY, VOFFLINE, UAENOSPC, and similar errors.

Correct a direct i/o code path which could overwrite a fileserver returned error code with a local RXGEN_CC_UNMARSHAL error.

Correct the StoreMini code path which could overwrite a fileserver returned error code with a local RXGEN_CC_UNMARSHAL error.

Ensure the rx call object is not locked when writing to the network socket.

Removed all knowledge of the KERNEL global lock from RX. Acquiring the GLOCK from RX is never safe if any other lock is held. Doing so is a lock order violation that can result in deadlocks.

Fixed a race in the opr_reservation system that could produce a cache entry reference undercount.

If a directory hash chain contains a circular link, a buffer page reference could be leaked for each traversal.

Each AFS3 directory header and page header contains a magic tag value that can be used in a consistency check but was not previously checked before use of each header. If the header memory is zero filled during a lookup, the search would fail producing an ENOENT error. Starting with this release the magic tag values are validated on each use. An EIO error is returned if there is a tag mismatch.

When renaming over a directory, the link count on the removed directory must drop from two to zero; not two to one.

Do not inadvertently report support for renameat2() flags.

When processing a direct-I/O truncate, be sure to update the inode size value.

"fs setcrypt -crypt auth" is now a permitted value. The "auth" mode provides integrity protection but no privacy protection.

Add new "aklog -levels " option which permits requesting "clear" and "auth" modes for use with yfs-rxgk.

New to v2021.04 (22 April 2021)

The Linux cache manager changes are primarily bug fixes for issues that have been present for years.

A possibility of an infinite kernel loop if a rare file write / truncate pattern occurs.

A bug in silly rename handling that can prevent cache manager initiated garbage collection of vnodes.

The potential of an overwritten ERESTARTSYS error during fetch or store data rpcs could result in transient failures.

New to v0.209 (13 March 2021)

Introduces support for Linux mainline 5.11 and 5.12 kernels.

Updated support for Linux ppc64 and ppc64le.

New bos getfile command compatible with AuriStorFS v0.209 and later bosserver. bos getfile is similar to bos getlog except that it can be used to fetch files containing arbitrary binary content.

fs setserverprefs and fs getserverprefs updated to support IPv6 and CIDR specifications.
Endpoint priorities can be set via the /etc/yfs/yfs-client.conf configuration file:
[afsd] endpoint-priorities = { 10.0.0.0/8 = 20000 10.10.10.10 = 30000 server.your-cell-name.com = 10000 2002::1234:abcd:ffff:c0a8:101/64 = 20000 }
The network specification may be an IP address, hostname or CIDR style network range specification. The priority is an integer, with the same meaning as the server ranks passed to fs setserverprefs. The default setting is to have no server priority information.

A generalised framework for the cache manager to execute userspace upcalls has been introduced. In the yfs_upcall framework, the kernel module asks the "afsd" process to create threads (as needed) to perform various services on behalf of the kernel module. The kernel module queues work for these threads which return to userspace, perform the tasks, deliver the result to the kernel module, and await the next task. The yfs_upcall framework tracks threads so it can ensure that all threads exit and return to userspace during shutdown.

Replaced the fragile legacy load/startup/shutdown/unload procedures with a new kernel module lifecycle. There are 6 module states defined, and 3 functions which drive the lifecycle.
yfs_standby() is the first thing called in the module by "afsd" when it starts up. It does whatever configuration is necessary to transition from a freshly loaded module to one that can begin accepting configuration syscalls from "afsd".
yfs_go() is called after "afsd" has completed loading configuration into the kernel module. Upon completion the module is ready to accept its first mount() request.
yfs_shutdown() controls the entire shutdown process. It eventually returns the kernel module to the freshly loaded state allowing a new yfs_standby() to start things back up again.

Switch to using the new background thread system for handling data stores so they can return early, if requested by the user, to userspace. This change uses the new opr_defer mechanism to detect stores where the user has specified a level of asynchrony. Once the requested quantity of data has been stored, and the fileserver has indicated via the user status mechanism that it has accepted the store, an opr_defer object is signalled. This allows the calling thread to return control to the user whilst the store completes in the background. Any errors which occur after this point are discarded.

Add interruptible versions of cv_wait and cv_timedwait that when signaled returns an error to be passed up to the syscall handler. Special care must be taken on Linux to handle fake signals such as those generated by systemtap and kprobe. Fake signals can not be blocked with signal masks.

Add versions of cv_timedwait that implement relative time waits instead of absolute time waits. Relative time waits are unaffected by system clock adjustments.

Prevent many (but not all) memory leaks at module unload.

Add a cache to store volume name to ID lookup results. This will cache both positive and negative lookups. This caching layer is located in front of the existing volume location information infrastructure.

Add sysctl statistics for the volname cache

fs.auristorfs.volname.errors = 0 fs.auristorfs.volname.hits_negative = 0 fs.auristorfs.volname.hits_positive = 16895 fs.auristorfs.volname.misses = 1032 fs.auristorfs.volname.waits = 0

Add volname caching control via sysctl

fs.auristorfs.volname.enabled_negative = 0 fs.auristorfs.volname.enabled_positive = 1

Plug leaks of RX_CALL_DEAD (301056), RXKADEXPIRED (19270409) and RXGKEXPIRED (1233242885) errors which were mapped to EIO instead of ETIMEDOUT.

Prefer local errors to rx_call errors. Do not blindly overwrite local errors that might be EINTR or ERESTARTSYS. EINTR or ERESTARTSYS must be passed to userspace without translation.

Do not permit RXGEN_CC_UNMARSHAL errors to take precedence over fileserver abort codes. Overwriting VBUSY and VOFFLINE abort codes prevent failover to alternate .readonly volume sites.

Since v0.176 a check for all volume sites being "offline" was disabled resulting in the generation of ETIMEDOUT error codes in place of ENODEV.

When multiple RPCs are in-flight for the same volume and fail with VOFFLINE or VSALVAGE error code, the volume status for that site could be set to an invalid out of range value. This could leave a volume site inaccessible until the kernel module is restarted.

The search for a valid server to issue a call to could be short circuited if an empty server slot was encountered. This could result in a failure to issue a call when additional sites are available.

If an RPC fails with VNOVOL or VMOVED, query the location service for updated volume location information only once per VFS operation instead of once for each received error.

Fixed a bug in afs_FlushActiveVcaches() that dates to IBM days. VBUSY and related state information was cleared inside the afs_Analyze() loop which broke failover. This function is called once a minute by the daemon thread.

Change the management of afs_Analyze loop state information to ensure that busy volume state and network error state is not carried forward from one afs_Analyze loop to another when a VFS operation requires multiple RPCs.

Prevent ERESTARTSYS or EINTR or ENOMEM errors from marking a fileserver down when querying a server's capabilities.

Replace the client processing of VOFFLINE and VSALVAGE errors that queries the location service and relies on persistent state to determine failover decisions with the VBUSY failover mechanism. The volume would become inaccessible if a VOFFLINE or VSALVAGE error was received from all sites before the ten minute daemon state reset. This was the reason for the unwritten rule that volumes should not be released more frequently than once every fifteen minutes. The new fallover logic can support a "vos release" every few seconds.

Use the full 64-bit data version for "localhero" directory updates. Directories whose data version grew beyond 2^31 would be assigned a truncated data version causing the directory to be fetched from the fileserver after each modification.

Prevented a race when populating the result of directory FetchStatus queries performed without holding the directory vcache lock. The race could result in out of date directory status information overwriting the status information protected by a callback promise.

Migrate all FetchStatus management to use the YFSFetchStatus data structure instead of down converting to the AFSFetchStatus structure. The YFSFetchStatus structure supports higher resolution time and additional metadata fields.

Starting with IBM AFS 3.2, IBM disabled use of an AFS mount point's 'mvid' field which (when valid) specified the volume id of the target volume. Ignoring the 'mvid' value requires that the target volume be recalculated each time the mount point is traversed. It is believed that IBM disabled the use of 'mvid' because its value could not be trusted. In AuriStorFS the validity of the 'mvid' value can be trusted and its use is once again enabled.

Introduce the yfs_priorities framework as a replacement for the legacy server preferences. The new framework supports CIDR specifications for both ipv4 and ipv6. With CIDR rules, priorities can be assigned by subnet instead of requiring individual assignments for each and every fileserver and vlserver endpoint.

Replace all of the volume location service query logic with the ubik_client based yfs_cell framework used by vos, fileserver, volserver, and salvageserver. The use of yfs_cell replaces a fragile and racy code base with a robust well-exercised thread safe code base.
A secondary goal of this replacement is the simplification of afs_Analyze() by removing all of the VL and RXGK error handling.

Replace token management with a thread-safe reference counted set of immutable objects.

Prevent a pathological thrashing scenario that can occur if the data cache is close to the threshold where it will start performing partial writes to the fileserver. Say that partial writing is triggered when N chunks are dirty, and that process A opens a file, dirties N - 1 chunks, and leaves the file open. Process B comes along and starts writing a large file. Every page written will make the dirty count N, and a partial write will be done to clean it, storing the entire single dirty chunk back to the server, with the dirty count going back to N -1. With a 1MB chunk size for example, this will result in doing 256 RPC calls to the server, storing roughly 128MB of data, instead of a single 1MB store.

Avoid unnecessary drops of the GLOCK during VBUSY / VOFFLINE retry processing that can prevent allocation of a rx connection when multiple calls are in flight to the same volume and each receive VBUSY / VOFFLINE errors.

Since v0.196, bulkstat queries that fail due to an inability to communicate with a fileserver hosting an online volume replica would produce an EIO error instead of ETIMEDOUT.

Fix a connection leak introduced in v0.189 that can be triggered by an failure to create a security class or if an attempt to perform a RXGK_AFSCombineTokens call to the location service fails.

When expanding a connection vector and security class creation fails fallback to an existing connection instead of failing. Its better to block and wait for a call slot to become available then to fail the vfs operation with RX_CALL_DEAD mapped to ETIMEDOUT.

Permit connection vector expansion in cases where connection vector creation is prohibited.

If afs_Analyze() is called without a connection structure it means that no RPC was issued to the fileserver. Therefore, there is no justification for discarding any callback promise. Discarding the callback will require status to be fetched from the fileserver.

The prior opr_cv_wait and opr_cv_timedwait kernel code was not freezable. If a machine was suspended whilst waiting in one of these functions, the kernel will complain with an error similar to

[254288.907204] Freezing user space processes ... [254308.909224] Freezing of tasks failed after 20.001 seconds (2 tasks refusing to freeze, wq_busy=0):
Make the wait loop freezable by, in kernels that support it, using freezable_schedule() or freezable_schedule_timeout(). In old kernels, add wrapper functions which add a call to try_to_freeze() to schedule.
The kernel's assumption is that a user-space process which has been frozen will restart its system call. This leads to a further issue - the freezer fakes a signal to all user threads. This signal means that schedule() returns immediately, and the module would busy wait, as the sequence number never changes. To fix this, attempt to clear the signal with recalc_sigpending() when returning from the freezer.
Finally, if after recalculating there is still a pending signal bail with ERESTARTSYS. This should never happen if signals have been blocked, or are running from a kernel thread, but it avoids busy waiting.

Add hardware accelerated cryptographic routines for __aarch64__.

systemtap, at least as implemented on RHEL 8, can result in "fake" signals interrupting threads. When this occurs, the thread will believe that there's a pending signal, but recalculating the pending state will clear the signal, as there is no real signal there. Having all/most signals blocked does not prevent this from happening.
Fake signals can trigger a couple of issues:

The module can return -ERESTARTSYS to the vfs with no actual pending signal. Depending on the particular syscall, this can result in leaking the error to userspace. This can occur if the fake signal appears between calls to splice_direct_to_actor(), or inside that function but before it had a chance to successfully transfer any data.

If the fake signal occurs while in splice_direct_to_actor(), and some data was successfully transferred, the splice operation will terminate early, and we'll get a "short splice". The splice succeeds so there is no immediate error to return, but the call will eventually fail with RXGEN_SS_UNMARSHAL because the amount of data sent to the fileserver will be less then what was advertized.

The dentry revalidate "bad volume parent" logic has been broken since the introduction of Linux support. As a result the volume parent was recalculated each time the volume root directory was accessed. The "volume parent" calculation and the check have been corrected.

Fail filesystem mount with ENXIO if "afsd" is not running.

Warn when the auristorfs kernel module converts an error code which is out of range for the Linux VFS to EIO.

Mapping out of range error code XXXX to EIO
The warning has been added to assist in identifying the source of leaking error codes that result in unexpected EIO errors. If this warning is observed in the kernel message log please notify AuriStor support.

During call startup treat ICMP unreachable errors as fatal. This permits new calls to fail fast.

Permit ICMP/ICMPv6 errors to terminate challenge events.

Update multi_Select() to block signals when building for Linux kernels to prevent overwriting ERESTARTSYS and EINTR errors.

rx_SetArrivalProc() must notify the call if the call is already in an error state. Otherwise, the installed arrival procedure will never be executed resulting in a deadlock.

Use relative time condvars to prevent clock modifications from impacting the rx event queue processing.

When searching for an rx connection by direction, epoch and cid also include the securityIndex value. This change reverts an IBM AFS 3.5 hack to prevent the fileserver from crashing. AuriStorFS filesrevers are not susceptible.

udebug: restore use of process names (vlserver, ptserver, budbserver) as alternatives to port service names.

Fix the default "fs setserverprefs" list to be fileserver instead of vlserver.

Default cell name changed to "your-cell-name.com".

Default yfs-client.conf now includes an "includedir /etc/yfs/yfs-client.conf.d" line.

During installation the AuriStorFS rpm disabled the "afs" SELinux module. Whne uninstalling AuriStorFS the "afs" SELinux module should be re-enabled.

cellservdb.conf updated openstack.org and bu.edu cell information.

New to v0.200 (4 November 2020)

Introduces support for Fedora Core 33 and Linux mainline 5.10 kernels.

The network path between a client and a server often traverses one or more network segments separated by NAT/PAT devices. If a NAT/PAT times out a RPCs endpoint translation mid-call, this can result in an extended delay before failure and the server being marked down, or worse, a call that never terminates and a client that appears to hang until the fileserver is restarted.

This release includes significant changes to the RX stack and the UNIX cache manager to detect such conditions, fail the calls quickly and detect when it is safe to retry the RPC.

NAT/PAT devices that drop endpoint mappings while in use are anti-social and can result in unwanted delays and even data loss. Whenever possible they should be avoided when they can be. That said, the changes in this release are a huge step to making the loss of endpoint mappings tolerable.

Fix segmentation fault of Backgrounder when krb5_get_credentials() fails due to lack of network connectivity.

Fix the "afsd" rxbind option which was ignored if the default port, 7001, is in use by another process on the system.

If a direct i/o StoreData or FetchData RPC failed such that it must be retried, the retried RPC would fail due to an attempt to Fetch or Store the wrong amount of data. This is fixed.

Servers are no longer marked down if RPCs fail with RX_CALL_PEER_RESET, RX_CALL_EXCEEDS_WINDOW, or RX_PROTOCOL_ERROR. RPCs that are safe to retry are retried.

Fixed a race between a call entering error state and call completion that can result in a the call remaining in the DALLY state and the connection channel remaining in use. If this occurs during process or system shutdown it can result in a deadlock.

During shutdown cancel any pending delayed aborts to prevent a potential deadlock. If a deadlock occurs when unloading a kernel module a reboot will be required.

Updated cellservdb.conf

The network path between a client and a server often traverses one or more network segments separated by NAT/PAT devices. If a NAT/PAT times out a RPCs endpoint translation mid-call, this can result in an extended delay before failure and the server being marked down, or worse, a call that never terminates and a client that appears to hang until the fileserver is restarted.

This release includes significant changes to the RX stack and the UNIX cache manager to detect such conditions, fail the calls quickly and detect when it is safe to retry the RPC.

NAT/PAT devices that drop endpoint mappings while in use are anti-social and can result in unwanted delays and even data loss. Whenever possible they should be avoided when they can be. That said, the changes in this release are a huge step to making the loss of endpoint mappings tolerable.

Fix segmentation fault of Backgrounder when krb5_get_credentials() fails due to lack of network connectivity.

Fix the "afsd" rxbind option which was ignored if the default port, 7001, is in use by another process on the system.

If a direct i/o StoreData or FetchData RPC failed such that it must be retried, the retried RPC would fail due to an attempt to Fetch or Store the wrong amount of data. This is fixed.

Servers are no longer marked down if RPCs fail with RX_CALL_PEER_RESET, RX_CALL_EXCEEDS_WINDOW, or RX_PROTOCOL_ERROR. RPCs that are safe to retry are retried.

Fixed a race between a call entering error state and call completion that can result in a the call remaining in the DALLY state and the connection channel remaining in use. If this occurs during process or system shutdown it can result in a deadlock.

During shutdown cancel any pending delayed aborts to prevent a potential deadlock. If a deadlock occurs when unloading a kernel module a reboot will be required.

If a server unreachable network error occurs during a direct i/o readpage bypass operation, it is possible for either a page to be improperly zero filled or for a general protection fault to occur. If a general protection fault doesn't occur, the kernel module will fail to unload due to a leaked rx call reference.

Fix mount source option processing. When SELinux is disabled on recent Linux mainline kernels mounting of /afs by "afsd" would fail unless the yfs-client.conf includes
[afsd] mountopts =

Updated cellservdb.conf

New to v0.197 (26 August 2020) and v0.198 (10 October 2020)

A new callback management framework for UNIX cache managers reduces the expense of processing volume callback RPCs from O(number of vcache objects) to O(1). A significant amount of lock contention has been avoided. The new design reduces the risk of the single callback service worker thread blocking. Delays in processing callbacks on a client can adversely impact fileserver performance and other clients in the cell.

More aggressive use of Bulk fetch status RPCs permit optimistic caching of vnode status information without additional round-trips. Individual fetch status RPCs are no longer issued if a bulk status fails to obtain the required status information.

Hardware accelerated crypto is once available for RHEL (and derivative) kernels. AuriStor's proprietary aes256-cts-hmac-sha1-96 and aes256-cts-hmac-sha512-384 implementations leverage Intel processor extensions: AESNI AVX2 AVX SSE41 SSSE3 to achieve the fastest encrypt, decrypt, sign and verify times for RX packets.

Support for Linux 5.8 and 5.9 mainline kernels.

Introduce support for SELinux mount options via
[afsd] mountopts =
The default mountopts value is
system_u:object_r:nfs_t:s0

Introduction of a CentOS specific repository

Pioctl support for Linux FUSE permits use of FUSE for authenticated /afs access.

This release optimizes the removal of "._" files that are used to store extended attributes by avoiding unnecessary status fetches when the directory entry is going to be removed.

When removing the final directory entry for an in-use vnode, the directory entry must be silly renamed on the fileserver to prevent removal of the backing vnode. The prior implementation resulted in wasted cycles searching for an unused name.

Behavior change! When the vfs performs a lookup on ".", immediately return the current vnode.

if the object is a mount point, do not perform fakestat and attempt to resolve the target volume root vnode.

do not perform any additional access checks on the vnode. If the caller already knows the vnode the access checks were performed earlier. If the access rights have changed, they will be enforced when the vnode is used just as they would have if the lookup of "." was performed within the vfs.

do not perform a fetch status or fetch data rpcs. Again, the same as if the lookup of "." was performed within the vfs.

Volumes mounted at more than one location in the /afs namespace are problematic on more than one operating system that do not expect directories to have more than one parent. It is particularly problematic if a volume is mounted within itself. Starting with this release any attempt to traverse a mountpoint to the volume containing the mountpoint will fail with ENODEV.

When generating the output displayed by /proc/fs/yfs/cellservdb.conf, generate fake server names and output ipv6 endpoints
servers = { afsdb1.cellname.invalid = { address = endpoint-as-string } }
Previously, the server name was an IPv4 address and if the endpoint was IPv6 the server name 0.0.0.0 would be generated.

Introduce enforcement of Linux file size limits. This fixes xfstests generic/394.

v0.196 was not publicly released.

New to v0.195 (14 May 2020)

This is a CRITICAL update for AuriStorFS Linux clients.

In Sep 2019 AuriStorFS v0.189 was released which provided faster and less CPU intensive writing of (>64GB) large files to /afs. These improvements introduced a hash collision bug in the store data path of the UNIX cache manager which can result in file corruption. If a hash collision occurs between two or more files that are actively being written to via cached I/O (not direct I/O), dirty data can be discarded from the auristorfs cache before it is written to the fileserver creating a file with a range of zeros (a hole) on the fileserver. This hole might not be visible to the application that wrote the data because the lost data was cached by the operating system. This bug has been fixed in v0.195 and it is for this reason that v0.195 has been designated a CRITICAL release for UNIX/Linux clients.

The Apr 2020 release of AuriStorFS v0.190 unmasked system error codes and propagated them to the vfs on many code paths. These changes re-introduced the possibility on Linux of an application receiving a SIGBUS signal if a non-fatal signal is delivered to the application while fetching pages for a memory mapped file. This has been fixed in the v0.195 release.

While debugging the Linux SIGBUS issue, it was observed that receipt of an ICMP network error in response to a transmitted packet could result in termination of an unrelated rx call and could mark a server down. If the terminated call is a StoreData RPC, permanent data loss will occur. All Linux clients derived from the IBM AFS code base experience this bug. The v0.195 release prevents this behavior.

This release includes changes that impact all supported UNIX/Linux cache managers. However, only on Linux can the number of allocated vcache entries grow without bounds subject to the memory limitations of the system. As of v0.195, Linux workflows that allocated tens of millions of vcache entries with prior releases now consistently reduce the allocation to the configured "stat" target value.

The auristor-repo-recommended-2-1.noarch.rpm AuriStorFS repository rpms include new functionality for CentOS clients. CentOS client systems use the same kernel modules as Red Hat Enterprise Linux client systems. However, the availability of new kernels is delayed; sometimes by many weeks. During the window where a new RHEL kernel has shipped and the CentOS kernel has not, attempts to update the AuriStorFS kernel module would fail as the latest available AuriStorFS kmod had no matching kernel package. The new repo rpm redirects CentOS systems to an alternate repository database that only lists AuriStorFS kmods for which a CentOS kernel is published. Note that CentOS regularly purges out of date kernels from their repositories. As a result, out of date AuriStorFS kmods will not be available once AuriStor's CentOS repository database has been synchronized.

Linux 5.7 kernel support

New to v0.194 (3 April 2020)

This is a CRITICAL update for AuriStorFS Linux clients but especially for clients deployed on RHELv6 system on which "systemtap" is in use.

AuriStorFS releases between v0.171 and v0.192 included a bug that could result in corrupted cache content for locally modified directories.

Executing "systemtap" on RHELv6 could result in system panic or data corruption when storing data to the fileserver.

The vcache has been increased from 10,000 to 150,000 entries.

/proc/sys/yfs/ has been replaced with /proc/sys/fs/auristorfs/.

Support for Linux 5.6 kernels and RHEL 7.8 has been introduced.

The /afs/.:mount/ syntax for accessing volumes by cell name and volume name or for accessing volume root directories conflicted with RPATH and Java CLASSPATH. Starting with the v0.193 release, the /afs/.:mount/ prefix has been replaced by /afs/.@mount/. [Red Hat Bugzilla 1794083]

New to v0.192 (30 January 2020)

The changes improve stability, efficiency, and scalability. Post-0.189 changes exposed race conditions and reference count errors which can lead to a system panic or deadlock. In addition to addressing these deficiencies this release removes bottlenecks that restricted the number of simultaneous vfs operations that could be processed by the auristorfs cache manager. The changes in this release have been successfully tested with greater than 400 simultaneous requests sustained for for several days.

New to v0.191 (16 December 2019)

Restore keyed cache manager functionality broken in v0.189.

Alter the processing of @sys directory entry evaluation to prevent the creation of Linux dentry aliases.

(RHEL7 only): Kernels for the 957 series (7.6) starting from 957.21.2, and all 1062 series (7.7) backported a change to d_splice_alias() that introduces an EIO error and inode reference leak if a connected alias is discovered when looking up a directory. This can result in two user visible symptoms:

EIO errors when looking up directories or mountpoints that have multiple paths, as can happen when evaluating @sys and the corresponding path substitution.

"VFS: Busy inodes after unmount" logged when /afs is unmounted.

This bug only impacts AuriStorFS when "[afsd] exportnfs" is enabled. This release handles EIO errors from affected kernels. [RedHat-BZ: 1749390, 1781158, 1781159]

Specify that "yfs" is an alias for "fs-auristorfs" which will allow the module to be automatically loaded when a mount request for filesystem type "auristorfs" is executed.

"afsd" now checks the initialization state and will error out if the initialization state is unexpected. This can prevent multiple instances of "afsd" from being launched.

Add support for multiple mounts of the afs root.

Alter shutdown processing to permit repeated mount and unmount of /afs. Unmounting /afs no longer results in termination of the "afsd" process.

[RHEL5] Fix shutdown for RHEL5.

Linux mainline kernel 5.0 removed the export of __kernel_fpu_begin and __kernel_fpu_end which are used to permit safe use of SIMD extensions. This change was backported to kernel 4.19.38 (released 2019-05-02) via d4ff57d0320bf441ad5a3084b3adbba4da1d79f8 and kernel 4.14.120 (released 2019-05-16) via a725c5201f0807a9f843db525f5f98f6c7a4c25b. Without these exports AuriStorFS must disable the use of SIMD extensions in the kernel module. Unfortunately, the build logic also disabled the use of SIMD extensions in userland. This release restores the use of SIMD extensions in userland. This fix impacts Fedora, Debian, Ubuntu and other non-RHEL based distributions.

[Red Hat] update the spec file to install firewalld service configuration in /usr/lib/firewalld/services/ instead of /usr/lib64/firewalld/services/ on 64-bit platforms.

[Debian / Ubuntu] fix build of pam_afs_session packages to include pam_afs_session and related binaries instead of placeholders.

Kernel module bug fixes.

New to v0.190 (14 November 2019)

Short-circuit busy volume retries after volume or volume location entry is removed.

Kernel module bug fixes.

New to v0.189 (28 October 2019)

Faster "git status" operation on repositories stored in /afs.

Faster and less CPU intensive writing of (>64GB) large files to /afs. Prior to this release writing files larger than 1TB might not complete. With this release store data throughput is consistent regardless of file size. (See "UNIX Cache Manager large file performance improvements" later in this file).

NFS export improvements.

Support for Linux kernel 5.3

New platforms: Fedora 31 and Oracle Linux 7

A race that might produce invalid negative directory entries was eliminated.

New to v0.188 (23 June 2019)

Automatically rehash unhashed dentry objects to prevent warnings or failures from shells, mount --bind, container orchestration systems, and other applications.

Increased clock resolution for timed waits from 1s to 1ns

Added error handling for rx multi rpcs interrupted by signals

New to v0.186 (29 May 2019)

Red Hat Enterprise Linux 8 and Fedora 30 now supported

Fedora 28 deprecated

v0.184 moved the /etc/yfs/cmstate.dat file to /var/yfs. With this change afsd would fail to start if /etc/yfs/cmstate.dat exists but contains invalid state information. This is fixed.

v0.184 introduced a potential deadlock during directory processing. This is fixed.

Many sites have noticed that clients with v0.184 installed might log Lost contact with xxxx server ... referencing a strange negative error code and that fileservers might log FetchData Write failure ... errors from any Linux client version.
These errors might correlate to corruption of pages in the Linux page cache. The corruption is that one or more contiguous pages might be inappropriately zero filled.
This release implements many code changes intended prevent Linux page cache are AFS disk cache corruption.

Better data version checks

More invalidation of cache chunk data version when zapping

Only zero fill pages past the server end of file

Always advance RPC stream pointer when skipping over missing pages or when populating pages from the disk cache chunk.

Never match a data version number equal to -1.

Avoid truncation races between find_get_page() and page locking.

Some sites have experienced failures of Linux mount --bind of /afs paths or getcwd returning ENOENT. This release fixes a dentry race that can produce an unhashed directory entry.
Some uses of the directory will continue to work, as the first lookup following the race will associate a new dentry with the inode, as an additional alias. Directories are not supposed to have aliases on Linux, so the vfs code assumes that d_alias is at most a list of 1 element, and accesses the entry in a slightly different way in a few places. Some sites get the new hashed dentry, others get the original unhashed one.

Propagate EINTR and ERESTARTSYS during location server queries to userland.

Handle common error table errors obtained outside an afs_Analyze loop. Map VL errors to ENODEV and RX, RXKAD, RXGK errors to ETIMEDOUT

Log all server down and server up events. Transition events from server probes failed to log messages.

Avoid leaking local errors to the fileserver if a failure occurs during Direct IO processing.

RX RPC networking:

If the RPC initiator successfully completes a call without consuming all of the response data fail the call by sending an RX_PROTOCOL_ERROR ABORT to the acceptor and returning a new error, RX_CALL_PREMATURE_END, to the initiator.
Prior to this change failure to consume all of the response data would be silently ignored by the initiator and the acceptor might resend the unconsumed data until any idle timeout expired. The default idle timeout is 60 seconds.

Avoid event cancellation race with rx call termination during process shutdown. This race when lost can prevent a process such as vos from terminating after successfully completing its work.

Avoid transmitting ABORT, CHALLENGE, and RESPONSE packets with an uninitialized sequence number. The sequence number is ignored for these packets but set it to zero.

New to v0.184 (26 March 2019)

The initial congestion window has been reduced from 10 Rx packets to 4. Packet reordering and loss has been observed when sending 10 Rx packets via sendmmsg() in a single burst. The lack of udp packet pacing can also increase the likelihood of transmission stalls due to ack clock variation.

Add support for the Linux 5.0 and 5.1 kernels.

The UNIX Cache Manager underwent major revisions to improve the end user experience by revealing more error codes, improving directory cache efficiency, and overall resiliency. The cache manager implementation was redesigned to be more compatible with operating systems such as Linux and macOS that support restartable system calls. With these changes errors such as "Operation not permitted", "No space left on device", "Quota exceeded", and "Interrupted system call" can be reliably reported to applications. Previously such errors might have been converted to "I/O error". These changes are expected to reduce the likelihood of "mount --bind" and getcwd failures with "No such file or directory" errors.

New to v0.179 and v0.180 (9 November 2018)

RX reliability and performance improvements for high latency and/or lossy network paths such as public wide area networks.

New to v0.170 (27 April 2018)

Support for RHEL 7.5 Final

Faster processing of cell configuration information by caching service name to port information.

RX call sequence number rollover to permit calls that require the transmission of more than 5.5TB of data.

Command parser Daylight Saving Time bug fix

Fix a bug that prevented immediate access to a mount point created with "fs mkmount" on the same machine.

Fix the setting of "[afsd] sysnames = " during cache manager startup.

New to v0.168

Corrects "fs setacl -negative" processing [CVE-2018-7168]

Adds support for Red Hat Enterprise Linux 7.5

Improved reliability for keyed cache managers. More persistent key acquisition renewals.

Major refresh to cellservdb.conf contents.

DNS SRV and DNS AFSDB records now take precedence when use_dns = yes

Kerberos realm hinting provided by
kerberos_realm = [REALM]

DNS host names are resolved instead of reliance on hard coded IP addresses
The cache manager now defaults to sparse dynamic root behavior. Only thiscell and those cells that are assigned aliases are included in /afs directory enumeration at startup. Other cells will be dynamically added upon first access.
Several other quality control improvements.

New to v0.167

Addresses a critical remote denial of service vulnerability [CVE-2017-17432]
Alters the volume location information expiration policy to reduce the risk of single points of failures after volume release operations.
'fs setquota' when issued with quota values larger than 2TB will fail against OpenAFS and IBM AFS file servers
Memory management improvements for the memory caches.

New to v0.164

When 'systemtap' is used to measure the lifetime of syscalls the 'afsd' upcall thread would begin to spin in a tight loop. This release adds restart support to the upcall thread.
Direct I/O support has been re-implemented. The prior implementation could result in use of unpinned memory pages. If a page was swapped to disk while in use the system would panic.
Cache Bypass support has been re-implemented. The new design can be implemented for other operating systems in the future. The Linux implementation leverages cached data (when present) for read operations but otherwise uncached pages are fetched directly from the fileserver without caching them in the AFS cache.
Read-ahead (pre-fetching) support expanded from 128KB to 4MB. Prefetch data is stored first to the Linux page cache and then back-filled to the AFS data cache. These changes should result in a noticeable improvement when reading data from the fileserver.
The AuriStorFS cache manager can now be installed side-by-side with kafs.
Memory and Disk caches now share much more common code. The relative performance of each is now easier to compare. Memory caches are much better supported.
In prior versions a crash could occur if the server list for a volume was modified while a new Rx connection object was in the process of being allocated and configured. This release includes a workaround to prevent the crash.
An optimization has been added when storing segments to short circuit data cache hash bucket scanning. It is expected this change will result in faster performance when storing small files.

New to v0.163

Linux 4.14. kernel support

New to v0.160

Support for Red Hat Enterprise Linux 7.4 3.10.0-693 and later kernels
Fix inconsistent "fs rmmount" behavior.
Removed all support for IBM DFS.
Add support for exporting /afs anonymously via NFS4, NFS3, and NFS4.
Reduced memory requirements for Rx Listener threads when Rx Jumbograms is disabled (the default).

New to v0.159

Various improvements to Direct IO read/write functionality
Improved VBUSY / VRESTARTING failover behavior.
Linux 4.13 kernel support

New to v0.157

Major reductions in resource contention resulting in improved parallel processing. Simultaneously accessing /afs from all cores on a 64-core system is "no big deal".
"vos" support for volume quotas larger than 2TB.
"fs flushvolume" works
A panic could occur during server capability testing

New to v0.150

Improved behavior when IPv6 is disabled
AuriStorFS file server detection improvements

New to v0.147

Public AuriStorFS client repository for Red Hat Enterprise, CentOS and Fedora Linux

Features:

Supports the /afs file namespace served by all AuriStorFS and OpenAFS cells.
IPv6 support

Known issues:

The AuriStor File System client requires SELinux permissive mode

macOS Installer (15.0 Sequoia)

Release Notes

Known Issues

If the Kerberos default realm is not configured, a delay of 6m 59s can occur before the AuriStorFS Backgrounder will acquire tokens and display its icon in the macOS menu. This is the result of macOS performing a Bonjour (MDNS) query in an attempt to discover the local realm.
"aklog" will fail to acquire tokens if the current working directory is located in an inaccessible AFS volume. When the "cwd" is inaccessible, the Heimdal framework is unable to initialize the Kerberos v5 context and reports an out of memory error.

New v2021.05-64 (22 Aug 2025)

The v2021.05-64 release fixes a race condition which can lead to a kernel panic when executing the "unlog" command.

v2021.05-63 (31 May 2025)

When AuriStorFS cache managers added support for per-file ACLs in v0.1, the masking of access rights based upon the inode's S_IRWXU mode bits were skipped when reading objects from AuriStorFS fileservers. The masking continued to be applied when reading objects from OpenAFS fileservers. This release applies S_IRWXU mode bits masking of granted access rights to both AuriStorFS and OpenAFS fileservers.
The existing fake stat mountpoint evaluation logic assumes that the fileserver returned a AFSFetchStatus.ParentVnode that refers to the parent directory from which the mountpoint was located. However, this is only guaranteed to be true if the mountpoint does not have a per-file ACL (which is permitted) and has never been cross-directory hard linked (which is also permitted).
This release improves the error handling logic to fail the fake stat evaluation if the AFSFetchStatus.ParentVnode does not match expectations.
This release fixes a metadata inconsistency when a mountpoint is created when disconnected mode is enabled. A mountpoint should be reported to the vfs as a directory and not a symlink.
The cache manager must track for each inode whether the object's status infomration was received from an AFS3 fileserver or an AuriStorFS fileserver. This release alters when that information is recorded to ensure that it is recorded each time the inode's status information is updated by a fileserver response.
This knowledge is required to properly determine how to interpret the status information.
Directory entries referring to mountpoints and volume roots must not be stored within the Directory Name Lookup Cache (DNLC). Instead, full lookups must be performed each time to ensure proper mountpoint resolution.
Prior to this release locally created mountpoints were stored into the DNLC by mistake.

v2021.05-62 (24 April 2025)

macOS Sequoia v15.4 vs Cisco AnyConnect VPN

Beginning with macOS 15.4, use of Cisco AnyConnect VPN breaks DNS SRV, AFSDB and TXT record lookups performed using the cross-platform DNS resolver API. As a result, clients which rely upon DNS-based zero-configuration resolution of Kerberos realms and/or AFS cells database services are unable to access those resources. This release switches the macOS builds to use the Apple DNS API.
The lookup failures impacted all command line tools, clients and services.
Note: Communication failures can occur if the VPN is configured by the organization to block the transmission of fragmented UDP packets. Set [afsd] rxmaxmtu = 1328 to ensure all RX packets are small enough to traverse the VPN link.
This release improves consistency of vnode metadata interpretation when the status information is received via an RPC which does not return a callback.

v2021.05-61 (31 March 2025)

Instead of allocating the same sized hash table for the vcache and dcache objects regardless of how many elements are expected to be stored, scale the hash table sizes. The vcache hash table will scale from 4096 buckets to 131072 buckets with a target of 4 entries per bucket. The dcache hash tables will scale from 4096 buckets to 65536 buckets with a target of 2 entries per bucket.
Historically, afs cache managers which create symlinks (including mountpoints) locally allocate a fake infinite lifetime callback to permit applications to use the symlink (or mountpoint) without requiring FetchStatus RPCs to be issued to the fileserver. This approach has drawbacks because it means that fileservers will not issue a callback break when the volume containing the symlink is taken offline, moved, deleted, etc. It also means that a callback will not be sent if another cache manager removes the symlink.
Beginning with this release the AuriStorFS cache manager will no longer allocate a fake infinite lifetime callback and will instead rely upon fileserver allocated callbacks.
"fs makemount" is supposed to create a mountpoint even if the target volume does not exist or is inaccessible. v2021.05 introduced support for creating mountpoints which kafs which broke the ability to create mountpoints to non-existent volumes.

v2021.05-60 (6 March 2025)

The cache manager retries fileserver RPCs up to 100 times when the fileserver replies with VBUSY, VOFFLINE or VSALVAGE. The cache manager fails the application request after the limit is reached. Prior to this release, the failure would result in the cache manager returning an ENODEV error to the application. But ENODEV indicates that the required volume does not exist and the fileserver has not indicated that the volume does not exist. The cache manager will now return EWOULDBLOCK to the application because the volume might be temporarily inaccessible.
Many OpenAFS and AuriStorFS fileservers can return VNOVOL as a side effect of a race between a volume going offline so that control can be given to the Volume Management Service (volserver) and an incoming RPC requesting that the volume be attached. A VNOVOL can also be returned because the volume was temporarily deleted from the vice partition prior to being replaced by a new clone, or prior to the fileserver marking the volume group as having been moved to another fileserver.
The fileserver replying to an RPC with VNOVOL cannot be trusted to mean that the volume has been permanently removed unless it is matched by the Location Service reporting the Volume Location (VL) entry has been deleted. From now on when a VNOVOL response is received and the VL entry exists, the cache manager will return EWOULDBLOCK instead of ENODEV to the application.
Prior to this release, any VL error code returned from the Location Service would be interpreted as meaning that the requested VL entry does not exist. Starting with this release only the VL_NOENT error will be interpreted as the VL entry does not exist.
Restrict the internal use of the ENOENT error code so the only use of ENOENT is to indicate that a directory entry does not exist. Previously, ENOENT could be returned if a disk cache entry could not be allocated or found. A disk cache related ENOENT returned when processing a directory might have been confused for the requested directory entry not existing.
Restrict the internal use of the ENODEV error code so that the only meaning of ENODEV is that the required volume does not exist. Previously, ENODEV could be returned due to transient errors when failing to contact the Location Service.

v2021.05-59 (13 February 2025)

The fix for garbage collection of idle unixuser objects introduced in v2021.05-56 was incomplete. As a result, Rx client connections might not be garbage collected.

v2021.05-57 (9 February 2025)

No new changes for macOS.

v2021.05-56 (7 February 2025)

Minor bug fixes to the Rx RPC implementation
The fix for garbage collection of idle unixuser objects introduced in v2021.05-55 was incomplete. As a result, Rx client connections might not be garbage collected.

v2021.05-55 (23 January 2025)

Minor bug fixes to the Rx RPC implementation
During garbage collection of idle unixuser objects, any Rx client connections associated with the unixuser (uid or pag) that were in use by another thread would not have their cache manager reference dropped. As a result the Rx client connections could never be garbage collected which is problematic on systems which churn through many unixusers. The behavior has been changed so that unixusers objects with in use Rx connections will not be freed. This permits future attempts to garbage collect unixusers to do so only after the Rx connections can also be freed.
Although this misbehavior has been present for decades, it is only since v2021.05-52 that the problem has been observed. It is possible that the changes to reduce the holding of the GLOCK has created a window of opportunity for connections to be simultaneously in use during unixuser garbage collection.

v2021.05-54 (19 January 2025)

The UNIX cache manager changes continue AuriStor's efforts to address a report that was received at the end of September. The end user site uses an OpenAFS cell to distribute software for execution by IBM Spectrum LSF (load sharing facility) managed compute nodes. Each of the approximately one dozen compute nodes is configured with approximately 100 processor cores and 1TB of ram. Each job scheduled by LSF runs in a newly allocated Process Authentication Group (PAG). Some jobs run for hours or days; others complete in a few seconds. LSF has been observed scheduling more than 3000 jobs in under a minute with peak counts of cache manager user objects at 1300. Everything ran well until it didn't. After some unexplained trigger event, all of the remaining free memory on the system would be consumed in about an hour causing the system to panic.

Over the last five months AuriStor has diligently worked to develop theories, optimize the AuriStorFS UNIX cache manager to address each theory; then move on to subsequent ideas once prior theories were disproved. Over this time we have replaced the memory management lifecycle for access rights entries and rx events. We have made it possible for Linux permission checks to be processed without acquiring the GLOCK in the majority of access patterns thereby increasing parallelism. The lifecycle management of PAG allocation and user entity tracking has been rewritten to avoid the GLOCK. The Linux inode lifecycle management has been rewritten which some sites attribute to elimination of swapping on their systems. All of these changes increased throughput and parallelism within and without the cache manager. However, none of them prevented the memory consumption once the unknown trigger event occurred.

It is with great pleasure that we can report that the trigger event and the source of the memory consumption have been identified. The trigger is a damaged Rx packet header which causes the AuriStor Rx stack to place an Rx connection into an RX_PROTOCOL_ERROR error state when there is an in-flight call to an OpenAFS fileserver. The memory consumption is due to the infinite loop that occurs when an Rx connection is in an error state and the error code is not one of the error codes which forces replacement of the Rx connection. Within each pass of the loop a structure is allocated to track the history of per-server errors so they can be logged when hard mount mode is enabled for the cell. AuriStor Rx permits up to 12 calls on each Rx connection. As a result there can be up to a dozen processes each spinning and allocating memory when the Rx connection is placed into the error state.

Days before Christmas 2023 AuriStor identified a race condition which can result in damaged Rx DATA packets being written to the network. Initial transmission of Rx DATA packets can be sent either from an application thread or an Rx listener. Packet retransmission occurs either in an Rx listener thread or in the Rx event thread. The race permits a second thread to modify the Rx packet headers while the sendmsg() syscall from the original transmission is executing. This race is possible because the original thread might exhaust its timeslice during the sendmsg() call and it might be many milliseconds before it is rescheduled on a busy system. In the meantime, packet retransmissions might be initiated either by an Rx listener thread processing an ACK packet or the Rx event thread processing a timeout event. If the initial packet transmission completes while the Rx headers are being rewritten the resulting packet sent on the wire might be garbage. AuriStor observed that if the modified packets were jumbograms that silent data corruption could occur so we disabled all use of jumbograms in v2021.05-35 We also notified both IBM and OpenAFS of the problem.

v2021.05-53 modified the RPC result analyze logic to force the replacement of Rx connections in an error state except when the error signals a token expiration. When a token is expired there is no benefit to replacing the connection until new tokens have been acquired. This release further ensures that infinite memory allocation cannot occur by tracking only the most recent error per fileserver; and only does so if hard mount mode is enabled for the cell.

Processes that issued syscalls which became stuck in a RPC retry loop could not be interrupted. In addition to the unintentional retry loops there are intentional retry loops such as those that occur because a fileserver repeated fails the call with a “VBUSY” error. “VBUSY” originally meant that the required volume was temporarily in use by the volume management service (“volserver”) and would be back in service soon so please wait. Beginning with IBM AFS 3.4 the error code was used to force a retry for reasons having nothing to do with the requested volume’s availability. IBM AFS 3.4 introduced the concept of multi-homed clients which meant that the fileserver could no longer track the identity of a cache manager by its IPv4 address. Instead, each cache manager randomly generates a UUID which the fileserver must query each time an incoming RPC is received on a previously unrecognized Rx connection. An Rx connection might be unrecognized because it’s new, or because the client’s IP address changed, or because the fileserver garbage collected its state information. Regardless of the reason, if a call is received and there is no Rx connection binding to a cache manager UUID, then the fileserver application worker thread to which the call was assigned must issue an outgoing RPC to the client’s RXAFSCB service. Each client has at most one worker thread. Prior to this release, every incoming RXAFSCB call required acquisition of the GLOCK before it could execute. If the GLOCK is unavailable, the RXAFSCB thread blocks and no other pending RXAFSCB calls can be processed. If a RXAFSCB call blocks for too long, the fileserver timeouts trigger and the fileserver replies to the incoming RXAFS call with “VBUSY”. “VBUSY” can also be sent if the fileserver’s quota on the number of calls from the same endpoint is exceeded.

The v2021.05-54 release removes the GLOCK requirement for many RXAFSCB RPCs including all of the UUID query RPCs and all whole volume callback RPCs. Callback RPCs for individual vnodes must still acquire the GLOCK. However, it is hoped that these changes will significantly reduce the time required to process the queries generated due to unrecognized Rx connections. The AuriStor Rx stack permits a dozen calls to be issued on a single logical connection which the fileserver sees as three unique Rx connections from the same endpoint.

Each Unix UID or PAG ID must use an independent Rx connection to each fileserver. On a system which processes /afs syscalls for 1200 UID or PAG IDs in a minute, there is the potential for 3600 Rx connections per fileserver. When the fileserver is OpenAFS, this quickly triggers per endpoint throttling and generates VBUSY storms which can result in unlucky client processes experiencing starvation. With this release, starving processes can be interrupted and RXAFSCB UUID queries can be completed without blocking.

AuriStor believes this release will provide significant benefits for large scale compute nodes, web servers and time sharing systems.

v2021.05-53 (4 January 2025)

v2021.05-52 switched the memory allocation strategy for rx_event objects from a global free list to reliance upon the operating system's memory allocator to allocate individual objects which are freed upon rx_event completion or cancelation. Reliance upon the system allocator introduces the risk of allocation failures due to out of memory conditions. If allocation of an rx_event object fails, a reference count leak on the associated rx_connection or rx_call object can occur preventing garbage collection of the object.
This release handles rx_event allocation failures:
- delayed ACK or delayed ABORT events are replaced by immediate transmission of the ACK or ABORT
- other cases fail the call or connection with an RX_OUT_OF_MEMORY error which is translated on the wire to RX_WIRE_CALL_DEAD (-1).
After executing a call to a fileserver, if the Rx connection is in an error state for any reason other than expired tokens, then schedule the Rx connection to be replaced the next time a call is attempted. All calls issued on an Rx connection that is already in an error state fail immediately preventing further communication with the associated fileserver from the UID/PAG.
Rx connections in an expired token error state are not replaced until new tokens are set.
Rx connections can be placed into an error state either due to local or remote failures. Such failures are often generated by the in-use security class (RXKAD/RXGK) but might be due to an out-of-memory condition, system clock advancement or reversal, or even application errors.
This release includes internal code changes to the management of servers, connections and access rights which will be used by a future release to separate PAG ID allocation from the system's UID namespace.
If 'afsd' is configured with a keytab, receipt of a PoweredOn event will force an attempt to acquire system tokens.
On supported macOS versions, enforce the use of code signing identities to authenticate mach messages involving the AFSBackgrounder and the AFSPreference panel.

v2021.05-52 (5 December 2024)

The v2021.05-50 replacement of the linked list implementation of the vcache access cache with a btree and last lookup cache introduced a memory corruption bug which is fixed in this release.
Eliminate the GLOCK requirement for the collection of function call and RPC statistics collection.
Replace small space allocation with a GLOCK free allocator.
Reimplement the cache manager tracing logging routines so they no longer require the GLOCK.
The Rx networking stack schedules many different types of time sensitive events associated with individual packet retransmission, delayed ack and abort processing, path mtu detection, keep alives, and other categories. Prior to this release, all memory allocated to Rx events objects would not be freed to the operating system until the userspace process terminated or the kernel module was unloaded.
Beginning with this release, instead of having a single, global, free list for rxevents, the system allocator is trusted to be able to allocate memory quickly. This simplifies the Rx implementation, and should be more efficient, as most system allocators will have per thread or core slabs, removing the contention of the global lock protecting the rxevent free list. In addition, the change protects against a network event allocating memory that cannot be readily freed.
On non-Total Store Ordering (TSO)architectures, relaxed atomics used as reference counters do not ensure completion of all prior stores or recency of later loads. This release replaces all read-modify-write atomics to be acquire-release, which, among other things, ensures that only the intended ordering is possible.

v2021.05-51 (26 November 2024)

Fix a vcache (aka inode) reference leak that can occur during an access rights query for a non-directory object if the neither the calling process nor the cache manager possesses tokens for the cell containing the object. The leak was introduced in v2021.05-45.
When scanning the token list for each user searching for expired tokens the user spinlock must be held to prevent races. The updated user management implementation pulled into v2021.05-45 failed to hold the spinlock.
During cache manager module shutdown reset global structures to the module initialization state. Doing so improves the likelihood that the module can be reinitialized.

v2021.05-50 (23 November 2024)

The linked list implementation of the vcache access rights cache was replaced with a btree and a last lookup cache. The btree design is more efficient for lookups and removals, and scales better as the number of uids and/or process authentication groups (pags) accessing /afs increases.
Prune access rights for expired users. Doing so reduces the cost of subsequent lookups.
Cache the access rights obtained for a uid|pag via the "anyAccess" field. Previously, a uid|pag that obtained its rights from the "anyAccess" field had to perform and fail a rights lookup on each and every vcache access.
When setting or clearing tokens for a single cell, no longer invalidate the access rights cached for the uid|pag for all cells. Instead, only invalidate the access rights cached for the cell whose tokens were set or cleared.

v2021.05-49 (16 November 2024)

The output of "tokens" command failed to report yfs-rxgk tokens was broken starting in v2021.05-46

v2021.05-48 (12 November 2024)

Preallocated buffer overflows in XDR responses (CVE-2024-10397)
The AuriStorFS and AFS3 RPC suites rely upon Sun RPC XDR to marshal binary data structures for network transfer. The AuriStor XDR implementation is derived from Sun Microsystems' Sun RPC code base. The Sun RPC XDR API permits memory for output parameters to (optionally) be preallocated which can result in various classes of memory corruption and/or memory leaks in RPC initiator processes.
The AuriStorFS v2021.05-48 release introduces additional data length validation checks within the AuriStor XDR implementation and prohibits the use of preallocated memory for string output parameters or fields. All cache managers, servers and command line tools are modified by these changes.

v2021.05-46 (28 October 2024)

Cache Manager:
- Prevent a kernel memory leak when server preferences are set via the yfs-client.conf [afsd] configuration or via "fs setserverprefs".
- Directory enumeration of a truncated directory now returns an error instead of assuming the end of the directory has been reached.
- Since AFS 3.0, the Unix cache manager has used the root identity credentials to create anonymous outgoing connections to the location service and each fileserver. However, if uid 0 is assigned a token, then those Rx connections will no longer be anonymous. Beginning with this release anonymous outgoing connections are always created with the NOPAG identity (uid 0xffffffff) instead of the root identity.
- When establishing an outgoing rxgk connection, do not fallback to the systemuser's credentials if the user's credentials resulted in a fatal error. Falling back to the systemuser's credentials can result in inappropriate use of an anonymous connection.
- Improved access rights cache correctness for YFS servers
  In prior releases, the access check logic used the file rights for any files fetched from an AuriStorFS fileserver. For files fetched from an AFS-3 fileserver (and, historically, for all files), it used the directory rights, with the (a)dmin right from the file mixed in. The (a)dmin right on a non-directory indicates that the object is owned by the authenticated user.
  This approach has some issues when combined with the access rights cache, and current fileserver callback behavior. On an AuriStorFS file server, the rights on a non-directory may be determined by the rights granted on its parent directory or, with per-file ACLs, those granted on the object itself. The fileserver will only break a non-directory's callback when a per-file ACL is changed - changing a directory ACL will not break callbacks on files within that directory. This means that changing a directory ACL will not invalidate access rights cache entries on files in tha directory, even if the effective ACL on this files has changed, and the cached rights are no longer correct.
  This release works around this by adding a new function which returns the access rights for a file hosted on an AuriStor fileserver. It uses the parent vnode information to locate the parent directory. If the parent directory isn't in the cache, or it doesn't have a valid callback, or if it has been changed since the file's access rights were cached, it clears the current access rights. Files without a parent directory must have per-file ACLs, and so their cached rights can be safely used.
  Note that files with parent vnodes may still have per-file ACLs, and that the breadcrumbing performed by the client may add parent vnode fields to vnodes which don't have them provided by the fileserver. Such vnodes may have their cached access rights cleared more frequently than necessary.
- Add a new mechanism for caching access rights within the vcache structure. This cache is protected via a vcache specific spinlock, and can be accessed without holding the GLOCK.
  This new cache mechanism returns the memory associated with cached rights back to the kernel's slab free memory pool instead of adding the unused rights structures to a cache manager managed free list. The previous cache implementation never returned allocated memory to the kernel. Instead, invalidated access rights were appended to a free access rights queue for later reuse.
- When a volume is accessed via multiple mountpoints a choice must be made regarding which mountpoint is considered to be the active (or parent) mountpoint. This release alters the behavior such that the active mountpoint is set every time a mountpoint is traversed.
  This behavior is easier to understand and is more likely to provide the expected result for a single process that repeatedly accesses volumes from multiple mountpoints. However, it can result in unexpected results when multiple processes are traversing multiple mountpoints in parallel without any synchronization.

v2021.05-45 (Not released)

v2021.05-44a (18 September 2024)

Authentication:
- AuriStorFS v2021.05-44 included an updated version of the Heimdal Kerberos framework used by AuriStorFS when acquiring yfs-rxgk and rxkad authentication tokens. The updated Heimdal included a bug which disabled the use of DNS SRV records for KDC discovery and DNS TXT records for realm discovery. As a side effect token acquisition might fail with an unable to reach any KDC in realm error. This is fixed in v2021.05-44a.

v2021.05-44 (17 August 2024)

Cache Manager:
- Since v0.192 the cache manager has failed acquire the global lock when upgrading a shared-lock to a write-lock during the execution of a background cache chunk file truncation.
Authentication:
- Neither MIT nor Heimdal gssapi nor their gss mechanisms consistently initialize the output 'minorStatus' parameter. Various functions can return either success or failure majorStatus values with minorStatus unassigned. As a result, stack garbage will be used when generating error messages. From now on libyfs_acquire will always initialize the minorStatus output variable to zero before calling into the gssapi library.
Command Parser:
- No longer accept the token "-" as a switch which eventually fails with a CMD_UNKNOWNSWITCH error. Instead, process the token as a data value.
- Optimize the processing of the loop which processes "source" command input.
- If the source command input file is "-", read from stdin.

v2021.05-41 (26 June 2024)

Rx Networking (libyfs_rx):
- A race during event creation can lead to the freeing of the event while its still in use.
- RFC1122 says that Net and Host unreachable ICMP errors might be transient and should therefore not be treated as fatal. There is no such language for the equivalent ICMPV6 errors. However, in practice ICMP6_DST_UNREACH_NOROUTE, ICMP6_DST_UNREACH_BEYONDSCOPE, and ICMP6_DST_UNREACH_ADDR can be transient.
  Linux has considered these ICMPV6 destination unreachable errors as non-fatal going back at least as far as the initial git repository commit.
  AuriStor Rx has always treated these as fatal errors which results in immediate termination of in-flight calls when received. Even if the network route corrects itself before the call timeout period expires. This release mirrors the Linux behavior and makes these errors non-fatal.
Cache Manager:
- For the first time the cache manager can detect the deletion of a volume and handle the creation of a new volume with the same name but a different volume id.
- If the location service reports the deletion of a volume, invalidate all mount points to that volume.
- RXAFS_GetCapabilities RPC failures should not be treated as a fatal error preventing failover to another replica site.
Authentication ("libyfs_acquire") used by aklog, vos, pts, bos, afsio:
- rxkad_k5 token acquisition krb5 ccache management. This release altered the krb5 credential cache management strategy once again to work around different bugs in MIT krb5 and Heimdal.
- New ACQUIRE_ERR_CRED_EXPIRED error code introduced to represent the case when a request for a service credential returns one that is already expired.
Command parser (libyfs_cmd):
- When parsing configuration files there is a depth limit of ten active inclusions. This limit was improperly enforced as a limit of ten included files instead of a depth of ten included files. As of this release it is now possible to populate an includedir directory with any number of .conf files.

v2021.05-40

Not released.

v2021.05-39 (20 May 2024)

Parallel Random Number Generation:
- AuriStorFS processes rely upon the krb5_generate_random() and RAND_bytes() functions to obtain random bytes for cryptographic operations and random counters. krb5_generate_random() internally acquires a mutex to protect internal state information. This mutex has become a significant barrier to the encryption and checksumming of Rx packets with both yfs-rxgk and rxkad.
  
  This release replaces general use of krb5_generate_random() and RAND_bytes() with a per-thread ChaCha20 CS-PRNG. This avoids the acquisition of a global mutex and permits increased parallelism on multi-core systems.
Rx Networking (libyfs_rx):
- The Rx network stack schedules a garbage collection operation to execute once per minute. This operation enforces call timeouts, destroys idle connections and destroys idle peers. The operation has historically been performed by the Rx event thread which is already responsible for performing actions in response to call RTOs, sending NAT Ping and keep-alive packets, and retrying connection challenge and reachability checks.
  
  The time complexity of the garbage collection operation is determined by the number of calls, connections, and peers. The busier the Rx endpoint the more work must be performed during each garbage collection run and the longer it takes to complete. While garbage collection is active other events cannot be processed which can interfere with the proper flow control of active calls.
  
  As with all Rx events, the garbage collection event is scheduled to execute at an absolute clock time. If the system clock drifts (or is administratively set) backwards garbage collection will not be performed until the clock catches up with the scheduled time.
  
  Another responsibility of the garbage collection procedure is to terminate calls if the system clock drifted backwards by five minutes or longer. However, when the clocked drifts backwards garbage collection is not performed until the clock has advanced beyond the point where calls require termination. As a result, calls are not terminated due to backwards clock drift and they can stall.
  
  This release re-implements the garbage collection procedure using a dedicated thread and relative waits. This change ensures that the garbage collection procedure will not prevent the execution of call related events and permits calls to be terminated when large backward clock drifts are detected.
Disk Cache Management:
- Since IBM AFS 3.5, the cache has been considered "too full" even if there exist cache files have been discarded but not yet truncated. When the cache is "too full" most operations that write to the cache will block until truncation of discarded cache files has been performed which results in unnecessary delays. This release fixes the cache such that that discarded but not yet truncated cache files do not block write operations.
- This release permits the cache truncation daemon thread to exit sooner if the cache manager is shutting down.
Improved failover when the RXGK service (co-located with each vlserver) fails to issue tokens. The failures might be the result of misconfiguration, an inability to read keys or loss of Ubik quorum.

v2021.05-38 (29 February 2024)

As with other AuriStorFS releases since the beginning of 2024, this release includes additional improvements to the Rx RPC implementation which are related to the possibility of silent data corruption when Rx jumbograms are in use. Prior releases disabled the negotiation of Rx jumbograms such that the v2021.05-37 Rx peer will refuse to send Rx jumbograms and will request that the remote peer does not send them. However, a bad actor could choose to send Rx jumbograms even though they were asked not to. v2021.05-38 introduces additional protections to ensure that a corrupt Rx jumbogram is dropped instead of being accepted.

The v2021.05-38 Rx RPC implementation also includes two optimizations. First, when Rx initiators complete a call they will no longer send an extra ACK packet to the Rx acceptor of the completed call. The sending of this unnecessary ACK creates additional work for the server which can result in increased latency for other calls being processed by the server.

Second, all AuriStor Rx services require a reach check for incoming calls from Rx peers to help protect against Distributed Reflection Denial of Service (DRDoS) attacks and execution of RPCs when the response cannot be delivered to the caller. A new reach check is required for each new call that arrives more than 60 seconds after the prior reach check completed. v2021.05-38 Rx considers the successful acknowledgment of a response DATA packet as a reach check validation. With this change reach checks will not be periodically required for a peer that completes at least one call per 60 seconds. A 1 RTT delay is therefore avoided each time a reach check can be avoided. In addition, reach checks require the service to process an additional ACK packet. Eliminating a large number of reach checks can improve overall service performance.

The final Rx RPC change in this release is specific to kernel implementations. Prior releases restricted the frequency of executing time scheduled Rx events to a granularity no smaller than 500ms. As a result an RTO timer event for a lost packet could not be shorter than 500ms even if the measured RTT for the connection is significantly smaller. The minimum RTO for a connection in AuriStor Rx is 200ms. The inability to schedule shorter timeouts impacts recovery from packet loss.

v2021.05-37 (5 February 2024)

Rx improvements:
- The v2021.05-36 release permanently disabled all use of Rx jumbograms due to a risk of silent data corruption. However, when advertising the number of acceptable datagrams in the ACK trailer a missing htonl() set the value to 16777216 instead of 1 on little-endian systems.
- When sending a PING ACK as a reachability test, ensure that the previousPacket field is properly assigned to the largest accepted DATA packet sequence number instead of zero.
- Replace the initialization state flag with two flags. One that indicates that Rx initialization began and the other that it succeeded. The first prevents multiple attempts at initialization after failure. The second prevents shutdown from accessing uninitialized structures if initialization failed.
Cache Manager Improvements:
- No longer refuse to start if both the 'cachedir' and 'memcache' options are present in the configuration file.
- New variable to store the maximum number of cache blocks used. which is accessible via /proc/fs/auristorfs/cache/blocks_used_max.

v2021.05-36 (10 January 2024)

Rx improvements:
- Permanently disable all use of Rx jumbograms due to a risk of silent data corruption.
  
  Ever since OpenAFS 1.0, and possibly before, a race condition has existed when Rx transmits packets. As the rx_call.lock is dropped when starting packet transmission, there is no protection for data that is being copied into the kernel by sendmsg(). It is critical that this packet data is not modified by another thread. However, races exist between the application, listener, and event threads that can lead to retransmissions starting whilst an original transmission is still in progress. This can lead to the packet headers being overwritten, and either the original transmission, the retransmission or both sending corrupt data to the peer.
  
  This corruption can affect the packet serial number or packet flags. It is particularly harmful when the packet flags are corrupted, as this can lead to multiple Rx packets which were intended to be sent as Rx jumbograms being delivered and misinterpreted as a single large packet. The eventual result of this depends on the Rx security class in play, but it can cause decrypt integrity errors (rxgk:crypt and rxgk:auth) or corruption of the data stream (rxnull, rxgk:clear or rxkad:auth).
  
  All AuriStorFS servers, OpenAFS 1.6 or later servers, and the Windows cache manager have been shipped with Rx jumbograms disabled by default. The UNIX cache managers however are shipped with jumbograms enabled. There are many AFS cells around the world that continue to deploy OpenAFS 1.4 or earlier fileservers which continue to negotiate the use of Rx jumbograms.
  
  It is worth noting that all AuriStorFS v0.198 and later fileservers and cache managers implement explicit checks that will recognize the corrupted application data stream and prevent corrupted file content from being stored either into an AFS vnode's backing store in the volume's object store or the cache manager's AFS cache. OpenAFS cache managers and fileservers do not have these protections.
  
  With Rx jumbograms disabled the maximum number of Rx packets in a datagram is reduced from 6 to 1; the maximum number of send and receive datagram fragments is reduced from 4 to 1; and the maximum advertised MTU is restricted to 1444 - the maximum rx packet size prior to the introduction of jumbograms in IBM AFS 3.5.
- If the rx call flow state transitions from either the RECOVERY or RESCUE states to the LOSS state as a result of an RTO resend event while writing packets to the network, cease transmission of any new DATA packets if there are packets in the resend queue.
- When the call flow state is LOSS and all packets in the resend queue have been retransmitted and yet the recovery point has not been reached, then permit new DATA packets to be sent in order to maintain a full congestion window.
- Add a safety check to prevent the estimated RTT from underflowing when the actual roundtrip time is smaller than 125us.
- Fix the computation of the padding required for rxgk encrypted packets. This bug results in packets sending 8 bytes fewer per packets than the network permits. This bug accidentally prevented the construction of Rx jumbograms when a call is protected by rxgk:crypt.
- Replace the random number generator with a more security source of random bytes.

v2021.05-33 (27 November 2023)

Rx improvements:
- Not all calls transfer enough data to be able to measure a smoothed round-trip time (SRTT). Calls which are unable to compute a SRTT should not be used to update the peer host RTO value which is used to initialize the RTO for subsequent calls.
  
  Without this change, a single DATA packet call will cause the peer host RTO to be reduced to 0ms. Subsequent calls will start with a RTO value of MAX(0, rxi_minPeerTimeout) where rxi_minPeerTimeout defaults to 200ms. If the actual measured RTO is greater than 200ms, then initial RTO will be too small resulting in premature triggering of the RTO timer and the call flow state entering the loss phase which can significantly hurt performance.
- Initialize the peer host RTO to rxi_minPeerTimeout (which defaults to 200ms) instead of one second. Although RFC6298 recommends the use of one second when no SRTT is available, Rx has long used the rxi_minPeerTimeout value for other purposes which are supposed to be consistent with initial RTO value. It should be noted that Linux TCP uses 200ms instead of one second for this purpose.
- If associating a security class with an Rx connection fails immediately place the Rx connection into an error state. A failure might occur if the security class is unable to access valid key material.
- If an incoming Rx call requires authentication and the security class is unable to successfully generate a challenge, put the incoming Rx connection into an error state and issue an abort to the caller.
- If an incoming Rx call requires authentication and the security class is able to generate a challenge but the challenge cannot be returned to Rx, then treat this as a transient error. Do not acknowledge the incoming DATA packet and do not place the Rx connection into an error state. An attempt to re-issue the challenge will be performed when the DATA packet is retransmitted.
- If an Rx call is terminated due to the expiration of the configured connection dead time, idle dead time, hard dead time, or as a result of clock drift, then send an ABORT to the peer notifying them that the call has been terminated. This is particularly important for terminated outgoing calls. If the peer does not know to terminate the call, then the call channel might be in use when the next outgoing call is issued using the same call channel. If the next incoming call is received by an in-use call channel, the receiver must drop the received DATA packet and return a BUSY packet. The call initiator will need to wait for a retransmission timeout to pass before retransmitting the DATA packet. Receipt of BUSY packets cannot be used to keep a call alive and therefore the requested call is at greater risk of timing out if the network path is congested.
aklog and krb5.log (via libyfs_acquire):
- If the linked Kerberos library implements krb5_cc_cache_match() and libacquire has been told to use an explicit principal name and credential cache, the Kerberos library might return KRB5_CC_NOTFOUND even though the requested credential cache is the correct one to use. This release will not call krb5_cc_cache_match() if the requested credential cache contains the requested principal.
Cell Service Database (cellservdb.conf):
- cellservdb.conf has been synchronized with the 31 Oct 2023 update to the grand.central.org CellServDB file.

v2021.05-32 (9 October 2023)

No significant changes for macOS compared to v2021.05-31

v2021.05-31 (25 September 2023)

New platform:
- macOS 14 Sonoma
macOS 14 Sonoma:

AuriStorFS v2021.05-29 and later installers for macOS 13 Ventura are compatible with macOS 14 Sonoma and do not need to be removed before upgrading to macOS 14 Sonoma. Installation of the macOS 14 Sonoma version of AuriStorFS is recommended.

Cache Manager:
- If an AuriStorFS cache manager is unable to use the yfs-rxgk security class when communicating with an AuriStorFS fileserver, it must assume it is IBM AFS 3.6 or OpenAFS and upgrade it to AuriStorFS if an upgrade probe returns a positive result. Once a fileserver's type is identified as AuriStorFS the type should never be reset; even if communication with the fileserver is lost or the fileserver restarts.
  
  If an AuriStorFS fileserver is replaced by an OpenAFS fileserver on the same endpoint, then the UUID of the OpenAFS must be different. As a result, the OpenAFS fileserver will be observed as distinct from the AuriStorFS fileserver that previously shared the endpoint.
  
  Prior to this release there were circumstances in which the cache manager discarded the fileserver type information and would fail to recognize the fileserver as an AuriStorFS fileserver when yfs-rxgk could not be used. This release prevents the cache manager from resetting the type information if the fileserver is marked down.
- If a fileserver's location service entry is updated with a new uniquifier value (aka version number), this indicates that one of the following might have changed:
  1. the fileserver's capabilities
  2. the fileserver's security policy
  3. the fileserver's knowledge of the cell-wide yfs-rxgk key
  4. the fileserver's endpoints
  Beginning with this release the cache manager will force the establishment of new Rx connections to the fileserver when the uniquifier changes. This ensures that the cache manager will attempt to fetch new per-fileserver yfs-rxgk tokens from the cell's RXGK service, enforce the latest security policy, and not end up in a situation where its existing tokens cannot be used to communicate with the fileserver.
aklog:
- Fix incorrect output when populating the server list for a service fails. The stashed extended error explaining the cause of the failure was not displayed.
- If a cell has neither _afs3-prserver._udp. DNS SRV records nor AFSDB records, the lookup of the cell's protection servers would fail if there is no local cell configuration details. The fallback to use _afs3-vlserver._udp. DNS SRV records did not work. This is corrected in this release.

v2021.05-30 (6 September 2023)

Do not mark a fileserver down in response to a KRB5 error code.
fs cleanacl must not store back to the file server a cleaned acl if it was inherited from a directory. Doing so will create a file acl.
Correct the generation of never expire rxkad_krb5 tokens from Kerberos v5 tickets which must have a start time of Unix epoch and an end time of 0xFFFFFFFF seconds. The incorrectly generated tokens were subject to the maximum lifetime of 30 days.
Correct the generation of the yfs-rxgk RESPONSE packet header which failed to specify the key version generation number used to encrypt the authenticator. If the actual key version is greater than zero, then the authenticator would fail to verify.
Enforce a maximum NAT ping period of 20s to ensure that NAT/PAT/firewall rules due not expire while Rx RPCs are in-flight.

v2021.05-29 (26 June 2023)

Execution of fs commands such as examine, whereis, listquota, fetchacl, cleanacl, storeacl, whoami, lsmount, bypassthreshold and getserverprefs could result in memory leaks by the AuriStorFS kernel extension.

v2021.05-27 (1 May 2023)

Fixes for bugs in vos introduced in v2021.05-26.

v2021.05-26 (17 April 2023)

Fixed a potential kernel memory leak when triggered by fs examine, fs listquota, or fs quota.
Increased logging of VBUSY, VOFFLINE, VSALVAGE, and RX_RESTARTING error responses. A log message is now generated whenever a task begins to wait as a result of one of these error responses from a fileserver. Previously, a message was only logged if the volume location information was expired or discarded.
Several changes to optimize internal volume lookups.
Faster failover to replica sites when a fileserver returns RX_RESTARTING, VNOVOL or VMOVED.
rxdebug regains the ability to report rx call flags and rx_connection flags.
The RXRPC library now terminates calls in the QUEUED state when an ABORT packet is received. This clears the call channel making it available to accept another call and reduces the work load on the worker thread pool.
Fileserver endpoint registration changes no longer result in local invalidation of callbacks from that server.
Receipt of an RXAFSCB_InitCallBackState3 RPC from a fileserver no longer resets the volume site status information for all volumes on all servers.

v2021.05-25 (28 December 2022)

The v2021.05-25 release includes further changes to RXRPC to improve reliability. The changes in this release prevent improper packet size growth. Packet size growth should never occur when a call is attempting to recover from packet loss; and is unsafe when the network path's maximum transmission unit is unknown. Packet size growth with be re-enabled in a future AuriStorFS release that includes Path MTU detection and the Extended SACK functionality.
Improved error text describing the source of invalid values in /etc/yfs/yfs-client.conf or included files and directories.

v2021.05-24 (25 October 2022)

New Platform: macOS 13 (Ventura)

v2021.05-22 (12 September 2022) and v2021.05-21 (6 September 2022)

RX RPC
- If receipt of a DATA packet causes an RX call to enter an error state, do not send the ACK of the DATA packet following the ABORT packet. Only send the ABORT packet.
- AuriStor RX has failed to count and report the number of RX BUSY packets that have been sent. Beginning with this change the sent RX BUSY packet count is once again included in the statistics retrieved via rxdebug server port -rxstats.
- Introduce minimum and maximum bounds checks on the ACK packet trailer fields. If the advertised values are out of bounds for the receiving RX stack, do not abort the call but adjust the values to be consistent with the local RX RPC implementation limits. These changes are necessary to handle broken RX RPC implementations or prevent manipulation by attackers.

New to v2021.05-20 (15 August 2022) and v2021.05-19 (13 August 2022)

RX RPC
- Include the DATA packet serial number in the transmitted reachability check PING ACK. This permits the reachability test ACK to be used for RTT measurement.
- Do not terminate a call due to an idle dead timeout if there is data pending in the receive queue when the timeout period expires. Instead deliver the received data to the application. This change prevents idle dead timeouts on slow lossy network paths.
- Fix assignment of RX DATA, CHALLENGE, and RESPONSE packet serial numbers in macOS (KERNEL). Due to a mistake in the implementation of atomic_add_and_read the wrong serial numbers were assigned to outgoing packets.

New to v2021.05-18 (12 July 2022)

Cache Manager
- Prevent a kernel memory leak of less than 64 bytes for each bulkstat RPC issued to a fileserver. Bulkstat RPCs can be frequently issued and over time this small leak can consume a large amount of kernel memory. Leak introduced in AuriStorFS v0.196.
- The Perl::AFS module directly executes pioctls via the OpenAFS compatibility pioctl interface instead of the AuriStorFS pioctl interface. When Perl::AFS is used to store an access control list (ACL), the deprecated RXAFS_StoreACL RPC would be used in place of the newer RXAFS_StoreACL2 or RXYFS_StoreOpaqueACL2 RPCs. This release alters the behavior of the cache manager to use the newer RPCs if available on the fileserver and fallback to the deprecated RPC. The use of the deprecated RPC was restricted to use of the OpenAFS pioctl interface.
RX RPC
- Handle a race during RX connection pool probes that could have resulted in the wrong RX Service ID being returned for a contacted service. Failure to identify that correct service id can result in a degradation of service.
- The Path MTU detection logic sends padded PING ACK packets and requests a PING_RESPONSE ACK be sent if received. This permits the sender of the PING to probe the maximum transmission unit of the path. Under some circumstances attempts were made to send negative padding which resulted in a failure when sending the PING ACK. As a result, the Path MTU could not be measured. This release prevents the use of negative padding.
Preparation for supporting macOS 13 Ventura when it is released in Fall 2022.
Some shells append a slash to an expanded directory name in response to tab completion. These trailing slashes interfered with "fs lsmount", "fs flushmount" and "fs removeacl" processing. This release includes a change to prevent these commands from breaking when presented a trailing slash.

New to v2021.05-17 (16 May 2022)

Cell Service Database Updates
- Update cern.ch, ics.muni.cz, ifh.de, cs.cmu.edu, qatar.cmu.edu, it.kth.se
- Remove uni-hohenheim.de, rz-uni-jena.de, mathematik.uni-stuttgart.de, stud.mathematik.uni-stuttgart.de, wam.umd.edu
- Add ee.cooper.edu
- Restore ams.cern.ch, md.kth.se, italia
Fix parsing of [afsd] rxwindow configuration which can be used to specified a non-default send/receive RX window size. The current default is 128 packets.
RX Updates
- Add nPacketsReflected

Logging improvements

Cache directory validation errors log messages now include the cache directory path.

Log the active configuration path if "debug" logging is enabled.

More details of rxgk token extraction failures.

New to v2021.05-16 (24 March 2022)

RX - Previous releases re-armed the Retransmission Timeout (RTO) each time a new unacknowledged packet is acknowledged instead of when a new leading edge packet is acknowledged. If leading edge data packet and its retransmission are lost, the call can remain in the "recovery" state where it continues to send new data packets until one of the following is true:
. the maximum window size is reached
. the number of lost and resent packets equals 'cwind'
at which point there is nothing left to transmit. The leading edge data packet can only be retransmitted when entering the "loss" state but since the RTO is reset with each acknowledged packet the call stalls for one RTO period after the last transmitted data packet is acknowledged.

This poor behavior is less noticiable with small window sizes and short lived calls. However, as window sizes and round-trip times increase the impact of a twice lost packet becomes significant.

RX - Never set the high-order bit of the Connection Epoch field. RX peers starting with IBM AFS 3.1b through AuriStor RX v0.191 ignore the source endpoint when matching incoming packets to RX connections if the high-order epoch bit is set. Ignoring the source endpoint is problematic because it can result in a call entering a zombie state whereby all PING ACK packets are immediately responded to the source endpoint of the PING ACK but any delayed ACK or DATA packets are sent to the endpoint bound to the RX connection. An RX client that moves from one network to another or which has a NAT|PAT device between it and the service can find themselves stuck.

Starting with AuriStor RX v0.192 the high-order bit is ignored by AuriStor RX peer when receiving packets. This change to always clear the bit prevents IBM AFS and OpenAFS peers from ignoring the source endpoint.

RX - The initial packetSize calculation for a call is altered to require that all constructed packets before the receipt of the first ACK packet are eligible for use in jumbograms if and only if the local RX stack has jumbograms enabled and the maximum MTU is large enough. By default jumbograms are disabled for all AuriStorFS services. This change will have a beneficial impact if jumbograms are enabled via configuration; or when testing RX performance with "rxperf".

New fs whereis -noresolve option displays the fileservers by network endpoint instead of DNS PTR record hostname.

New to v2021.05-15 (24 January 2022)

kernel - fixed YFS_RXGK service rx connection pool leak

New to v2021.05-14 (20 January 2022)

fs mkmount permit mount point target strings longer than 63 characters.
afsd enhance logging of yfs-rxgk token renewal errors.

afsd gains a "principal = configuration option for use with keytab acquisition of yfs-rxgk tokens for the cache manager identity.

kernel - Avoid unnecessary rx connection replacement by racing threads after token replacement or expiration.

kernel - Fix a regression introduced in v2021.05 where an anonymous combined identity yfs-rxgk token would be replaced after three minutes resulting in the connection switching from yfs-rxgk to rxnull.

kernel - Fix a regression introduced in v0.208 which prevented the invalidation of cached access rights in response to a fileserver callback rpc. The cache would be updated after the first FetchStatus rpc after invalidation.

kernel - Reset combined identity yfs-rxgk tokens when the system token is replaced.

kernel - The replacement of rx connection bundles in the cache manager to permit more than four simultaneous rx calls per uid/pag with trunked rx connections introduced the following regressions in v2021.05.

a memory leak of discarded rx connection objects

failure of NAT ping probes after replacement of an connection

inappropriate use of rx connections after a service upgrade failure

All of these regressions are fixed in patch 14.

New to v2021.05-12 (7 October 2021)

fs ignorelist -type afsmountdir in prior releases could prevent access to /afs.

Location server rpc timeout restored to two minutes instead of twenty minutes.

Location server reachability probe timeout restored to six seconds instead of fifty seconds.

Cell location server upcall results are now cached for fifteen seconds.

Multiple kernel threads waiting for updated cell location server reachability probes now share the results of a single probe.

RX RPC implementation lock hierarchy modified to prevent a lock inversion.

RX RPC client connection reference count leak fixed.

RX RPC deadlock during failed connection service upgrade attempt fixed..

New to v2021.05-9 (25 October 2021)

First public release for macOS 12 Monterey build using XCode 13. When upgrading macOS to Monterey from earlier macOS releases, please upgrade AuriStorFS to v2021.05-9 on the starting macOS release, upgrade to Monterey and then install the Monterey specific v2021.05-9 release.

Improved logging of "afsd" shutdown when "debug" mode is enabled.

Minor RX network stack improvements

New to v2021.05-3 (10 June 2021)

Fix for [cells] cellname = {...} without server list.

New to v2021.05 (31 May 2021)

Multi-homed location servers are finally managed as a single server instead of treating each endpoint as a separate server. The new functionality is a part of the wholesale replacement of the former cell management infrastructure. Location server communication is now entirely managed as a cluster of multi-homed servers for each cell. The new infrastructure does not rely upon the global lock for thread safety.

This release introduces a new infrastructure for managing user/pag entities and tracking their per cell tokens and related connection pools.

Expired tokens are no longer immediately deleted so that its possible for them to be listed by "tokens" for up to two hours.

Prevent a lock inversion introduced in v0.208 that can result in a deadlock involving the GLOCK and the rx call.lock. The deadlock can occur if a cell's list of location servers expires and during the rebuild an rx abort is issued.

Add support for rxkad "auth" mode rx connections in addition to "clear" and "crypt". "auth" mode provides integrity protection without privacy.

Add support for yfs-rxgk "clear" and "auth" rx connection modes.

Do not leak a directory buffer page reference when populating a directory page fails.

Re-initialize state when populating a disk cache entry using the fast path fails and a retry is performed using the slow path. If the data version changes between the attempts it is possible for truncated disk cache data to be treated as valid.

Log warnings if a directory lookup operation fails with an EIO error. An EIO error indicates that an invalid directory header, page header, or directory entry was found.

Do not overwrite RX errors with local errors during Direct-I/O and StoreMini operations. Doing so can result in loss of VBUSY, VOFFLINE, UAENOSPC, and similar errors.

Correct a direct i/o code path which could overwrite a fileserver returned error code with a local RXGEN_CC_UNMARSHAL error.

Correct the StoreMini code path which could overwrite a fileserver returned error code with a local RXGEN_CC_UNMARSHAL error.

Ensure the rx call object is not locked when writing to the network socket.

Removed all knowledge of the KERNEL global lock from RX. Acquiring the GLOCK from RX is never safe if any other lock is held. Doing so is a lock order violation that can result in deadlocks.

Fixed a race in the opr_reservation system that could produce a cache entry reference undercount.

If a directory hash chain contains a circular link, a buffer page reference could be leaked for each traversal.

Each AFS3 directory header and page header contains a magic tag value that can be used in a consistency check but was not previously checked before use of each header. If the header memory is zero filled during a lookup, the search would fail producing an ENOENT error. Starting with this release the magic tag values are validated on each use. An EIO error is returned if there is a tag mismatch.

"fs setcrypt -crypt auth" is now a permitted value. The "auth" mode provides integrity protection but no privacy protection.

Add new "aklog -levels " option which permits requesting "clear" and "auth" modes for use with yfs-rxgk.

Update MKShim to Apple OpenSource MITKerberosShim-79.

Report KLL errors via a notification instead of throwing an exception which (if not caught) will result in process termination.

If an exception occurs while executing "unlog" catch it and ignore it. Otherwise, the process will terminate.

New to v2021.04 (22 April 2021)

Primarily bug fixes for issues that have been present for years.

A possibility of an infinite kernel loop if a rare file write / truncate pattern occurs.

A bug in silly rename handling that can prevent cache manager initiated garbage collection of vnodes.

New to v0.209 (13 March 2021)

fs setserverprefs and fs getserverprefs updated to support IPv6 and CIDR specifications.

Improved error handling during fetch data and store data operations.

Prevents a race between two vfs operations on the same directory which can result in caching of out of date directory contents.

Use cached mount point target information instead of evaluating the mount point's target upon each access.

Avoid rare data cache thrashing condition.

Prevent infinite loop if a disk cache error occurs after the first page in a chunk is written.

Network errors are supposed to be returned to userspace as ETIMEDOUT. Previously some were returned as EIO.

When authentication tokens expire, reissue the fileserver request anonymously. If the anonymous user does not have permission either EACCES or EPERM will be returned as the error to userspace. Previously the vfs request would fail with an RXKADEXPIRED or RXGKEXPIRED error.

If growth of an existing connection vector fails, wait on a call slot in a previously created connection instead of failing the vfs request.

Volume and fileserver location query infrastructure has been replaced with a new modern implementation.

Replace the cache manager's token management infrastructure with a new modern implementation.

New to v0.206 (12 January 2021) - Bug fixes

Prevents a possible panic during unmount of /afs.

Improved failover and retry logic for offline volumes.

New to v0.205 (24 December 2020) - Bug fixes

Volume name-to-id cache improvements

Fix expiration of name-to-id cache entries

Control volume name-to-id via sysctl

Query volume name-to-id statistics via sysctl

Improve error handling for offline volumes

Fix installer to prevent unnecessary installation of Rosetta 2 on Apple Silicon

New to v0.204 (25 November 2020) - Bug fix for macOS Big Sur

v0.204 prevents a kernel panic on Big Sur when AuriStorFS is stopped and restarted without an operating system reboot.

introduces a volume name-to-id cache independent of the volume location cache.

New to v0.203 (13 November 2020) - Bug fix for macOS

v0.203 prevents a potential kernel panic due to network error.

New to v0.201 (12 November 2020) - Universal Big Sur (11.0) release for Apple Silicon and Intel

v0.201 introduces a new cache manager architecture on all macOS versions except for High Sierra (10.12). The new architecture includes a redesign of:

kernel extension load

kernel extension unload (not available on Big Sur)

/afs mount

/afs unmount

userspace networking

The conversion to userspace networking will have two user visible impacts for end users:
The Apple Firewall as configured by System Preferences -> Security & Privacy -> Firewall is now enforced. The "Automatically allow downloaded signed software to receive incoming connections" includes AuriStorFS.

Observed network throughput is likely to vary compared to previous releases.

On Catalina the "Legacy Kernel Extension" warnings that were displayed after boot with previous releases of AuriStorFS are no longer presented with v0.201.

AuriStorFS /afs access is expected to continue to function when upgrading from Mojave or Catalina to Big Sur. However, as AuriStorFS is built specifically for each macOS release, it is recommended that end users install a Big Sur specific AuriStorFS package.
AuriStorFS on Apple Silicon supports hardware accelerated aes256-cts-hmac-sha1-96 and aes128-cts-hmac-sha1-96 using AuriStor's proprietary implementation.

New to v0.200 (4 November 2020) - Final release for macOS El Capitan (10.11)

The network path between a client and a server often traverses one or more network segments separated by NAT/PAT devices. If a NAT/PAT times out a RPCs endpoint translation mid-call, this can result in an extended delay before failure and the server being marked down, or worse, a call that never terminates and a client that appears to hang until the fileserver is restarted.

This release includes significant changes to the RX stack and the UNIX cache manager to detect such conditions, fail the calls quickly and detect when it is safe to retry the RPC.

NAT/PAT devices that drop endpoint mappings while in use are anti-social and can result in unwanted delays and even data loss. Whenever possible they should be avoided when they can be. That said, the changes in this release are a huge step to making the loss of endpoint mappings tolerable.

Fix segmentation fault of Backgrounder when krb5_get_credentials() fails due to lack of network connectivity.

Fix the "afsd" rxbind option which was ignored if the default port, 7001, is in use by another process on the system.

If a direct i/o StoreData or FetchData RPC failed such that it must be retried, the retried RPC would fail due to an attempt to Fetch or Store the wrong amount of data. This is fixed.

Servers are no longer marked down if RPCs fail with RX_CALL_PEER_RESET, RX_CALL_EXCEEDS_WINDOW, or RX_PROTOCOL_ERROR. RPCs that are safe to retry are retried.

Fixed a race between a call entering error state and call completion that can result in a the call remaining in the DALLY state and the connection channel remaining in use. If this occurs during process or system shutdown it can result in a deadlock.

During shutdown cancel any pending delayed aborts to prevent a potential deadlock. If a deadlock occurs when unloading a kernel module a reboot will be required.

Updated cellservdb.conf

New to v0.197.1 (31 August 2020) and v0.198 (10 October 2020)

Prevent Dead vnode has core/unlinkedel/flock panic introduced in v0.197.

New to v0.197 (26 August 2020)

A new callback management framework for UNIX cache managers reduces the expense of processing volume callback RPCs from O(number of vcache objects) to O(1). A significant amount of lock contention has been avoided. The new design reduces the risk of the single callback service worker thread blocking. Delays in processing callbacks on a client can adversely impact fileserver performance and other clients in the cell.

Bulk fetch status RPCs are available on macOS for the first time. Bulk fetch status permits optimistic caching of vnode status information without additional round-trips. Individual fetch status RPCs are no longer issued if a bulk status fails to obtain the required status information.

Hardware accelerated crypto is now available for macOS cache managers. AuriStor's proprietary aes256-cts-hmac-sha1-96 and aes256-cts-hmac-sha512-384 implementations leverage Intel processor extensions: AESNI AVX2 AVX SSE41 SSSE3 to achieve the fastest encrypt, decrypt, sign and verify times for RX packets.

This release optimizes the removal of "._" files that are used to store extended attributes by avoiding unnecessary status fetches when the directory entry is going to be removed.

When removing the final directory entry for an in-use vnode, the directory entry must be silly renamed on the fileserver to prevent removal of the backing vnode. The prior implementation risked blindly renaming over an existing silly rename directory entry.

Behavior change! When the vfs performs a lookup on ".", immediately return the current vnode.

if the object is a mount point, do not perform fakestat and attempt to resolve the target volume root vnode.

do not perform any additional access checks on the vnode. If the caller already knows the vnode the access checks were performed earlier. If the access rights have changed, they will be enforced when the vnode is used just as they would have if the lookup of "." was performed within the vfs.

do not perform a fetch status or fetch data rpcs. Again, the same as if the lookup of "." was performed within the vfs.

Volumes mounted at more than one location in the /afs namespace are problematic on more than one operating system that do not expect directories to have more than one parent. It is particularly problematic if a volume is mounted within itself. Starting with this release any attempt to traverse a mountpoint to the volume containing the mountpoint will fail with ENODEV.

When evaluating volume root vnodes, ensure that the vnode's parent is set to the parent directory of the traversed mountpoint and not the mountpoint. Vnodes without a parent can cause spurious ENOENT errors on Mojave and later.

v0.196 was not publicly released.

New to v0.195 (14 May 2020)

This is a CRITICAL update for AuriStorFS macOS clients.

In Sep 2019 AuriStorFS v0.189 was released which provided faster and less CPU intensive writing of (>64GB) large files to /afs. These improvements introduced a hash collision bug in the store data path of the UNIX cache manager which can result in file corruption. If a hash collision occurs between two or more files that are actively being written to via cached I/O (not direct I/O), dirty data can be discarded from the auristorfs cache before it is written to the fileserver creating a file with a range of zeros (a hole) on the fileserver. This hole might not be visible to the application that wrote the data because the lost data was cached by the operating system. This bug has been fixed in v0.195 and it is for this reason that v0.195 has been designated a CRITICAL release for UNIX/Linux clients.

While debugging a Linux SIGBUS issue, it was observed that receipt of an ICMP network error in response to a transmitted packet could result in termination of an unrelated rx call and could mark a server down. If the terminated call is a StoreData RPC, permanent data loss will occur. All Linux clients derived from the IBM AFS code base experience this bug. The v0.195 release prevents this behavior.

This release includes changes that impact all supported UNIX/Linux cache managers. On macOS there is reduced lock contention between kernel threads when the vcache limit has been reached.

The directory name lookup cache (DNLC) implementation was replaced. The new implementation avoids the use of vcache pointers which did not have associated reference counts, and eliminates the invalidation overhead during callback processing. The DNLC now supports arbitrary directory name lengths; the prior implementation only cached entries with names not exceeding 31 characters.

Prevent matching arbitrary cell name prefixes as aliases. For example "/afs/y" should not be an alias for "your-file-system.com". Some shells, for example "zsh", query the filesystem for names as users type. Delays between typed characters result in filesystem lookups. When this occurs in the /afs dynroot directory, this could result in cellname prefix string matches and the dynamic creation of directory entries for those prefixes.

New to v0.194 (2 April 2020)

This is a CRITICAL release for all macOS users. All prior macOS clients whether AuriStorFS or OpenAFS included a bug that could result in data corruption either when reading or writing.

This release also fixes these other issues:

sign and notarize installer plugin "afscell" bundle. The lack of digital signature prevented the installer from prompting for a cellname on some macOS versions.

prevent potential for corruption when caching locally modified directories.

v0.193 was withdrawn due to a newly introduced bug that could result in data corruption.

New to v0.192 (30 January 2020)

The changes improve stability, efficiency, and scalability. Post-0.189 changes exposed race conditions and reference count errors which can lead to a system panic or deadlock. In addition to addressing these deficiencies this release removes bottlenecks that restricted the number of simultaneous vfs operations that could be processed by the AuriStorFS cache manager. The changes in this release have been successfully tested with greater than 400 simultaneous requests sustained for for several days.

New to v0.191 (16 December 2019)

Restore keyed cache manager capability broken in v0.189.

Add kernel module version string to AuriStorFS Preference Pane.

Other kernel module bug fixes.

New to v0.190 (14 November 2019)

Short-circuit busy volume retries after volume or volume location entry is removed.

New to v0.189 (28 October 2019)

Faster "git status" operation on repositories stored in /afs.

Faster and less CPU intensive writing of (>64GB) large files to /afs. Prior to this release writing files larger than 1TB might not complete. With this release store data throughput is consistent regardless of file size. (See "UNIX Cache Manager large file performance improvements" later in this file).

macOS Catalina (8 October 2019)

AuriStorFS v0.188 released for macOS Catalina (10.15)

New to v0.188 (23 June 2019)

Increased clock resolution for timed waits from 1s to 1ns

Added error handling for rx multi rpcs interrupted by signals

New to v0.186 (29 May 2019)

v0.184 moved the /etc/yfs/cmstate.dat file to /var/yfs. With this change afsd would fail to start if /etc/yfs/cmstate.dat exists but contains invalid state information. This is fixed.

v0.184 introduced a potential deadlock during directory processing. This is fixed.

Handle common error table errors obtained outside an afs_Analyze loop. Map VL errors to ENODEV and RX, RXKAD, RXGK errors to ETIMEDOUT

Log all server down and server up events. Transition events from server probes failed to log messages.

RX RPC networking:

If the RPC initiator successfully completes a call without consuming all of the response data fail the call by sending an RX_PROTOCOL_ERROR ABORT to the acceptor and returning a new error, RX_CALL_PREMATURE_END, to the initiator.
Prior to this change failure to consume all of the response data would be silently ignored by the initiator and the acceptor might resend the unconsumed data until any idle timeout expired. The default idle timeout is 60 seconds.

Avoid transmitting ABORT, CHALLENGE, and RESPONSE packets with an uninitialized sequence number. The sequence number is ignored for these packets but set it to zero.

New to v0.184 (26 March 2019)

The initial congestion window has been reduced from 10 Rx packets to 4. Packet reordering and loss has been observed when sending 10 Rx packets via sendmmsg() in a single burst. The lack of udp packet pacing can also increase the likelihood of transmission stalls due to ack clock variation.

The UNIX Cache Manager underwent major revisions to improve the end user experience by revealing more error codes, improving directory cache efficiency, and overall resiliency. The cache manager implementation was redesigned to be more compatible with operating systems such as Linux and macOS that support restartable system calls. With these changes errors such as "Operation not permitted", "No space left on device", "Quota exceeded", and "Interrupted system call" can be reliably reported to applications. Previously such errors might have been converted to "I/O error".

New to v0.180 (9 November 2018)

RX reliability and performance improvements for high latency and/or lossy network paths such as public wide area networks.

A fix for a macOS firewall triggered kernel panic introduced in v0.177.

New to v0.177 (17 October 2018)

A fix to AuriStor's RX implementation bug introduced in v0.176 that interferes with communication with OpenAFS and IBM Location and File Services.

New to v0.176 (3 October 2018)

AuriStor's RX implementation has undergone a major upgrade of its flow control model. Prior implementations were based on TCP Reno Congestion Control as documented in RFC5681; and SACK behavior that was loosely modelled on RFC2018. The new RX state machine implements SACK based loss recovery as documented in RFC6675, with elements of New Reno from RFC5682 on top of TCP-style congestion control elements as documented in RFC5681. The new RX also implements RFC2861 style congestion window validation.

When sending data the RX peer implementing these changes will be more likely to sustain the maximum available throughput while at the same time improving fairness towards competing network data flows. The improved estimation of available pipe capacity permits an increase in the default maximum window size from 60 packets (84.6 KB) to 128 packets (180.5 KB). The larger window size increases the per call theoretical maximum throughput on a 1ms RTT link from 693 mbit/sec to 1478 mbit/sec and on a 30ms RTT link from 23.1 mbit/sec to 49.39 mbit/sec.

Improve shutdown performance by refusing to give up callbacks to known unreachable file servers and apply a shorter timeout period for the rest.

Permit RXAFSCB_WhoAreYou to be successfully executed after an IBM AFS or OpenAFS fileserver unintentionally requests an RX service upgrade from RXAFSCB to RXYFSCB.

RXAFS timestamps are conveyed in unsigned 32-bit integers with a valid range of 1 Jan 1970 (Unix Epoch) through 07 Feb 2106. UNIX kernel timestamps are stored in 32-bit signed integers with a valid range of 13 Dec 1901 through 19 Jan 2038. This discrepency causes RXAFS timestamps within the 2038-2106 range to display as pre-Epoch.

RX Connection lifecycle management was susceptible to a number of race conditions that could result in assertion failures, the lack of a NAT ping connection to each file server, and the potential reuse of RX connections that should have been discarded.

This release includes a redesigned lifecycle that is thread safe, avoids assertions, prevents NAT ping connection loss, and ensures that discarded connections are not reused.

The 0.174 release unintentionally altered the data structure returned to xstat_cm queries. This release restores the correct wire format.

Since v0.171, if a FetchData RPC fails with a VBUSY error and there is only one reachable fileserver hosting the volume, then the VFS request will immediately with an ETIMEDOUT error ("Connection timed out").

v0.176 corrects three bugs that contributed to this failure condition. One was introduced in v0.171, another in 0.162 and the final one dates to IBM AFS 3.5p1.

The intended behavior is that a cache manager, when all volume sites fail an RPC with a VBUSY error, will sleep for up to 15 seconds and then retry the RPC as if the VBUSY error had never been received. If the RPC continues to receive VBUSY errors from all sites after 100 cycles, the request will be failed with EWOULDBLOCK ("Operation would block") and not ETIMEDOUT.

Prefer VOLMISSING and VOLBUSY error states to network error states when generating error codes to return to the VFS layer. This will result in ENODEV ("No such device") errors when all volume sites return VNOVOL or VOFFLINE errors and EWOULDBLOCK ("Operation would block") errors when all volume sites return VBUSY errors. (v0.176)

New to v0.174 (24 September 2018)

macOS Mojave (10.14) support

New to v0.170 (27 April 2018)

Faster processing of cell configuration information by caching service name to port information.

RX call sequence number rollover to permit calls that require the transmission of more than 5.5TB of data.

Command parser Daylight Saving Time bug fix

Fix a bug that prevented immediate access to a mount point created with "fs mkmount" on the same machine.

Fix the setting of "[afsd] sysnames = " during cache manager startup.

New to v0.168 (6 March 2018)

Corrects "fs setacl -negative" processing [CVE-2018-7168]

Improved reliability for keyed cache managers. More persistent key acquisition renewals.

Major refresh to cellservdb.conf contents.

DNS SRV and DNS AFSDB records now take precedence when use_dns = yes

Kerberos realm hinting provided by
kerberos_realm = [REALM]

DNS host names are resolved instead of reliance on hard coded IP addresses
The cache manager now defaults to sparse dynamic root behavior. Only thiscell and those cells that are assigned aliases are included in /afs directory enumeration at startup. Other cells will be dynamically added upon first access.
Several other quality control improvements.

New to v0.167 (7 December 2017)

Addresses a critical remote denial of service vulnerability [CVE-2017-17432]
Alters the volume location information expiration policy to reduce the risk of single points of failures after volume release operations.
'fs setquota' when issued with quota values larger than 2TB will fail against OpenAFS and IBM AFS file servers
Memory management improvements for the memory caches.

New to v0.164 (11 November 2017)

Internal cache manager redesign. No new functionality.

New to v0.160 (21 September 2017)

Support for OSX High Sierra's new Apple File System (APFS). Customers must upgrade to v0.160 or later before upgrading to OSX High Sierra.
Reduced memory requirements for rx listener thread
Avoid triggering a system panic if an AFS local disk cache file is deleted or becomes inaccessible.
Fixes to "fs" command line output

New to v0.159 (7 August 2017)

Improved failover behavior during volume maintenance operations
Corrected a race that could lead the rx listener thread to enter an infinite loop and cease processing incoming packets.

New to v0.157 (12 July 2017)

Bundled with Heimdal 7.4 to address CVE-2017-11103 (Orpheus' Lyre puts Kerberos to sleep!)
"vos" support for volume quotas larger than 2TB.
"fs flushvolume" works
Fixed a bug that can result in a system panic during server capability testing

New to v0.150

AuriStorFS file server detection improvements

New to v0.149

rxkad encryption is enabled by default. Use "fs setcrypt off" to disable encryption when tokens are available.
Fix a bug in atomic operations on Sierra and El Capitan which could adversely impact Rx behavior.

New to v0.128

Extended attribute ._ files are automatically removed when the associated files are unlinked
Throughput improvements when sending data

New to v0.121

OSX Sierra support

New to v0.117

Cache file moved to a persistent location on local disk
AuriStor File System graphics
Improvements in Background token fetch functionality
Fixed a bug introduced in v0.44 that could result in an operating system crash when enumerating AFS directories containing Unicode file names (v0.106)
El Capitan security changes prevented Finder from deleting files and directories. As of v0.106, the AuriStor OSX client implements the required functionality to permit the DesktopHelperService to securely access the AFS cache as the user permitting Finder to delete files and directories.

Features:

Not vulnerable to OPENAFS-SA-2015-007.
Office 2011 can save to /afs.
Office 2016 can now save files to /afs.
OSX Finder and Preview can open executable documents without triggering a "Corrupted File" warning. .AI, .PDF, .TIFF, .JPG, .DOCX, .XLSX, .PPTX, and other structured documents that might contain scripts were impacted.
All file names are now stored to the file server using Unicode UTF-8 Normalization Form C which is compatible with Microsoft Windows.
All file names are converted to Unicode UTF-8 Normalization Form D for processing by OSX applications.

Known issues:

None

macOS Installer (14.0 Sonoma)

Release Notes

Known Issues

If the Kerberos default realm is not configured, a delay of 6m 59s can occur before the AuriStorFS Backgrounder will acquire tokens and display its icon in the macOS menu. This is the result of macOS performing a Bonjour (MDNS) query in an attempt to discover the local realm.
"aklog" will fail to acquire tokens if the current working directory is located in an inaccessible AFS volume. When the "cwd" is inaccessible, the Heimdal framework is unable to initialize the Kerberos v5 context and reports an out of memory error.

New v2021.05-64 (22 Aug 2025)

The v2021.05-64 release fixes a race condition which can lead to a kernel panic when executing the "unlog" command.

v2021.05-63 (31 May 2025)

When AuriStorFS cache managers added support for per-file ACLs in v0.1, the masking of access rights based upon the inode's S_IRWXU mode bits were skipped when reading objects from AuriStorFS fileservers. The masking continued to be applied when reading objects from OpenAFS fileservers. This release applies S_IRWXU mode bits masking of granted access rights to both AuriStorFS and OpenAFS fileservers.
The existing fake stat mountpoint evaluation logic assumes that the fileserver returned a AFSFetchStatus.ParentVnode that refers to the parent directory from which the mountpoint was located. However, this is only guaranteed to be true if the mountpoint does not have a per-file ACL (which is permitted) and has never been cross-directory hard linked (which is also permitted).
This release improves the error handling logic to fail the fake stat evaluation if the AFSFetchStatus.ParentVnode does not match expectations.
This release fixes a metadata inconsistency when a mountpoint is created when disconnected mode is enabled. A mountpoint should be reported to the vfs as a directory and not a symlink.
The cache manager must track for each inode whether the object's status infomration was received from an AFS3 fileserver or an AuriStorFS fileserver. This release alters when that information is recorded to ensure that it is recorded each time the inode's status information is updated by a fileserver response.
This knowledge is required to properly determine how to interpret the status information.
Directory entries referring to mountpoints and volume roots must not be stored within the Directory Name Lookup Cache (DNLC). Instead, full lookups must be performed each time to ensure proper mountpoint resolution.
Prior to this release locally created mountpoints were stored into the DNLC by mistake.

v2021.05-62 (24 April 2025)

macOS Sequoia v15.4 vs Cisco AnyConnect VPN

Beginning with macOS 15.4, use of Cisco AnyConnect VPN breaks DNS SRV, AFSDB and TXT record lookups performed using the cross-platform DNS resolver API. As a result, clients which rely upon DNS-based zero-configuration resolution of Kerberos realms and/or AFS cells database services are unable to access those resources. This release switches the macOS builds to use the Apple DNS API.
The lookup failures impacted all command line tools, clients and services.
Note: Communication failures can occur if the VPN is configured by the organization to block the transmission of fragmented UDP packets. Set [afsd] rxmaxmtu = 1328 to ensure all RX packets are small enough to traverse the VPN link.
This release improves consistency of vnode metadata interpretation when the status information is received via an RPC which does not return a callback.

v2021.05-61 (31 March 2025)

Instead of allocating the same sized hash table for the vcache and dcache objects regardless of how many elements are expected to be stored, scale the hash table sizes. The vcache hash table will scale from 4096 buckets to 131072 buckets with a target of 4 entries per bucket. The dcache hash tables will scale from 4096 buckets to 65536 buckets with a target of 2 entries per bucket.
Historically, afs cache managers which create symlinks (including mountpoints) locally allocate a fake infinite lifetime callback to permit applications to use the symlink (or mountpoint) without requiring FetchStatus RPCs to be issued to the fileserver. This approach has drawbacks because it means that fileservers will not issue a callback break when the volume containing the symlink is taken offline, moved, deleted, etc. It also means that a callback will not be sent if another cache manager removes the symlink.
Beginning with this release the AuriStorFS cache manager will no longer allocate a fake infinite lifetime callback and will instead rely upon fileserver allocated callbacks.
"fs makemount" is supposed to create a mountpoint even if the target volume does not exist or is inaccessible. v2021.05 introduced support for creating mountpoints which kafs which broke the ability to create mountpoints to non-existent volumes.

v2021.05-60 (6 March 2025)

The cache manager retries fileserver RPCs up to 100 times when the fileserver replies with VBUSY, VOFFLINE or VSALVAGE. The cache manager fails the application request after the limit is reached. Prior to this release, the failure would result in the cache manager returning an ENODEV error to the application. But ENODEV indicates that the required volume does not exist and the fileserver has not indicated that the volume does not exist. The cache manager will now return EWOULDBLOCK to the application because the volume might be temporarily inaccessible.
Many OpenAFS and AuriStorFS fileservers can return VNOVOL as a side effect of a race between a volume going offline so that control can be given to the Volume Management Service (volserver) and an incoming RPC requesting that the volume be attached. A VNOVOL can also be returned because the volume was temporarily deleted from the vice partition prior to being replaced by a new clone, or prior to the fileserver marking the volume group as having been moved to another fileserver.
The fileserver replying to an RPC with VNOVOL cannot be trusted to mean that the volume has been permanently removed unless it is matched by the Location Service reporting the Volume Location (VL) entry has been deleted. From now on when a VNOVOL response is received and the VL entry exists, the cache manager will return EWOULDBLOCK instead of ENODEV to the application.
Prior to this release, any VL error code returned from the Location Service would be interpreted as meaning that the requested VL entry does not exist. Starting with this release only the VL_NOENT error will be interpreted as the VL entry does not exist.
Restrict the internal use of the ENOENT error code so the only use of ENOENT is to indicate that a directory entry does not exist. Previously, ENOENT could be returned if a disk cache entry could not be allocated or found. A disk cache related ENOENT returned when processing a directory might have been confused for the requested directory entry not existing.
Restrict the internal use of the ENODEV error code so that the only meaning of ENODEV is that the required volume does not exist. Previously, ENODEV could be returned due to transient errors when failing to contact the Location Service.

v2021.05-59 (13 February 2025)

The fix for garbage collection of idle unixuser objects introduced in v2021.05-56 was incomplete. As a result, Rx client connections might not be garbage collected.

v2021.05-57 (9 February 2025)

No new changes for macOS.

v2021.05-56 (7 February 2025)

Minor bug fixes to the Rx RPC implementation
The fix for garbage collection of idle unixuser objects introduced in v2021.05-55 was incomplete. As a result, Rx client connections might not be garbage collected.

v2021.05-55 (23 January 2025)

Minor bug fixes to the Rx RPC implementation
During garbage collection of idle unixuser objects, any Rx client connections associated with the unixuser (uid or pag) that were in use by another thread would not have their cache manager reference dropped. As a result the Rx client connections could never be garbage collected which is problematic on systems which churn through many unixusers. The behavior has been changed so that unixusers objects with in use Rx connections will not be freed. This permits future attempts to garbage collect unixusers to do so only after the Rx connections can also be freed.
Although this misbehavior has been present for decades, it is only since v2021.05-52 that the problem has been observed. It is possible that the changes to reduce the holding of the GLOCK has created a window of opportunity for connections to be simultaneously in use during unixuser garbage collection.

v2021.05-54 (19 January 2025)

The UNIX cache manager changes continue AuriStor's efforts to address a report that was received at the end of September. The end user site uses an OpenAFS cell to distribute software for execution by IBM Spectrum LSF (load sharing facility) managed compute nodes. Each of the approximately one dozen compute nodes is configured with approximately 100 processor cores and 1TB of ram. Each job scheduled by LSF runs in a newly allocated Process Authentication Group (PAG). Some jobs run for hours or days; others complete in a few seconds. LSF has been observed scheduling more than 3000 jobs in under a minute with peak counts of cache manager user objects at 1300. Everything ran well until it didn't. After some unexplained trigger event, all of the remaining free memory on the system would be consumed in about an hour causing the system to panic.

Over the last five months AuriStor has diligently worked to develop theories, optimize the AuriStorFS UNIX cache manager to address each theory; then move on to subsequent ideas once prior theories were disproved. Over this time we have replaced the memory management lifecycle for access rights entries and rx events. We have made it possible for Linux permission checks to be processed without acquiring the GLOCK in the majority of access patterns thereby increasing parallelism. The lifecycle management of PAG allocation and user entity tracking has been rewritten to avoid the GLOCK. The Linux inode lifecycle management has been rewritten which some sites attribute to elimination of swapping on their systems. All of these changes increased throughput and parallelism within and without the cache manager. However, none of them prevented the memory consumption once the unknown trigger event occurred.

It is with great pleasure that we can report that the trigger event and the source of the memory consumption have been identified. The trigger is a damaged Rx packet header which causes the AuriStor Rx stack to place an Rx connection into an RX_PROTOCOL_ERROR error state when there is an in-flight call to an OpenAFS fileserver. The memory consumption is due to the infinite loop that occurs when an Rx connection is in an error state and the error code is not one of the error codes which forces replacement of the Rx connection. Within each pass of the loop a structure is allocated to track the history of per-server errors so they can be logged when hard mount mode is enabled for the cell. AuriStor Rx permits up to 12 calls on each Rx connection. As a result there can be up to a dozen processes each spinning and allocating memory when the Rx connection is placed into the error state.

Days before Christmas 2023 AuriStor identified a race condition which can result in damaged Rx DATA packets being written to the network. Initial transmission of Rx DATA packets can be sent either from an application thread or an Rx listener. Packet retransmission occurs either in an Rx listener thread or in the Rx event thread. The race permits a second thread to modify the Rx packet headers while the sendmsg() syscall from the original transmission is executing. This race is possible because the original thread might exhaust its timeslice during the sendmsg() call and it might be many milliseconds before it is rescheduled on a busy system. In the meantime, packet retransmissions might be initiated either by an Rx listener thread processing an ACK packet or the Rx event thread processing a timeout event. If the initial packet transmission completes while the Rx headers are being rewritten the resulting packet sent on the wire might be garbage. AuriStor observed that if the modified packets were jumbograms that silent data corruption could occur so we disabled all use of jumbograms in v2021.05-35 We also notified both IBM and OpenAFS of the problem.

v2021.05-53 modified the RPC result analyze logic to force the replacement of Rx connections in an error state except when the error signals a token expiration. When a token is expired there is no benefit to replacing the connection until new tokens have been acquired. This release further ensures that infinite memory allocation cannot occur by tracking only the most recent error per fileserver; and only does so if hard mount mode is enabled for the cell.

Processes that issued syscalls which became stuck in a RPC retry loop could not be interrupted. In addition to the unintentional retry loops there are intentional retry loops such as those that occur because a fileserver repeated fails the call with a “VBUSY” error. “VBUSY” originally meant that the required volume was temporarily in use by the volume management service (“volserver”) and would be back in service soon so please wait. Beginning with IBM AFS 3.4 the error code was used to force a retry for reasons having nothing to do with the requested volume’s availability. IBM AFS 3.4 introduced the concept of multi-homed clients which meant that the fileserver could no longer track the identity of a cache manager by its IPv4 address. Instead, each cache manager randomly generates a UUID which the fileserver must query each time an incoming RPC is received on a previously unrecognized Rx connection. An Rx connection might be unrecognized because it’s new, or because the client’s IP address changed, or because the fileserver garbage collected its state information. Regardless of the reason, if a call is received and there is no Rx connection binding to a cache manager UUID, then the fileserver application worker thread to which the call was assigned must issue an outgoing RPC to the client’s RXAFSCB service. Each client has at most one worker thread. Prior to this release, every incoming RXAFSCB call required acquisition of the GLOCK before it could execute. If the GLOCK is unavailable, the RXAFSCB thread blocks and no other pending RXAFSCB calls can be processed. If a RXAFSCB call blocks for too long, the fileserver timeouts trigger and the fileserver replies to the incoming RXAFS call with “VBUSY”. “VBUSY” can also be sent if the fileserver’s quota on the number of calls from the same endpoint is exceeded.

The v2021.05-54 release removes the GLOCK requirement for many RXAFSCB RPCs including all of the UUID query RPCs and all whole volume callback RPCs. Callback RPCs for individual vnodes must still acquire the GLOCK. However, it is hoped that these changes will significantly reduce the time required to process the queries generated due to unrecognized Rx connections. The AuriStor Rx stack permits a dozen calls to be issued on a single logical connection which the fileserver sees as three unique Rx connections from the same endpoint.

Each Unix UID or PAG ID must use an independent Rx connection to each fileserver. On a system which processes /afs syscalls for 1200 UID or PAG IDs in a minute, there is the potential for 3600 Rx connections per fileserver. When the fileserver is OpenAFS, this quickly triggers per endpoint throttling and generates VBUSY storms which can result in unlucky client processes experiencing starvation. With this release, starving processes can be interrupted and RXAFSCB UUID queries can be completed without blocking.

AuriStor believes this release will provide significant benefits for large scale compute nodes, web servers and time sharing systems.

v2021.05-53 (4 January 2025)

v2021.05-52 switched the memory allocation strategy for rx_event objects from a global free list to reliance upon the operating system's memory allocator to allocate individual objects which are freed upon rx_event completion or cancelation. Reliance upon the system allocator introduces the risk of allocation failures due to out of memory conditions. If allocation of an rx_event object fails, a reference count leak on the associated rx_connection or rx_call object can occur preventing garbage collection of the object.
This release handles rx_event allocation failures:
- delayed ACK or delayed ABORT events are replaced by immediate transmission of the ACK or ABORT
- other cases fail the call or connection with an RX_OUT_OF_MEMORY error which is translated on the wire to RX_WIRE_CALL_DEAD (-1).
After executing a call to a fileserver, if the Rx connection is in an error state for any reason other than expired tokens, then schedule the Rx connection to be replaced the next time a call is attempted. All calls issued on an Rx connection that is already in an error state fail immediately preventing further communication with the associated fileserver from the UID/PAG.
Rx connections in an expired token error state are not replaced until new tokens are set.
Rx connections can be placed into an error state either due to local or remote failures. Such failures are often generated by the in-use security class (RXKAD/RXGK) but might be due to an out-of-memory condition, system clock advancement or reversal, or even application errors.
This release includes internal code changes to the management of servers, connections and access rights which will be used by a future release to separate PAG ID allocation from the system's UID namespace.
If 'afsd' is configured with a keytab, receipt of a PoweredOn event will force an attempt to acquire system tokens.
On supported macOS versions, enforce the use of code signing identities to authenticate mach messages involving the AFSBackgrounder and the AFSPreference panel.

v2021.05-52 (5 December 2024)

The v2021.05-50 replacement of the linked list implementation of the vcache access cache with a btree and last lookup cache introduced a memory corruption bug which is fixed in this release.
Eliminate the GLOCK requirement for the collection of function call and RPC statistics collection.
Replace small space allocation with a GLOCK free allocator.
Reimplement the cache manager tracing logging routines so they no longer require the GLOCK.
The Rx networking stack schedules many different types of time sensitive events associated with individual packet retransmission, delayed ack and abort processing, path mtu detection, keep alives, and other categories. Prior to this release, all memory allocated to Rx events objects would not be freed to the operating system until the userspace process terminated or the kernel module was unloaded.
Beginning with this release, instead of having a single, global, free list for rxevents, the system allocator is trusted to be able to allocate memory quickly. This simplifies the Rx implementation, and should be more efficient, as most system allocators will have per thread or core slabs, removing the contention of the global lock protecting the rxevent free list. In addition, the change protects against a network event allocating memory that cannot be readily freed.
On non-Total Store Ordering (TSO)architectures, relaxed atomics used as reference counters do not ensure completion of all prior stores or recency of later loads. This release replaces all read-modify-write atomics to be acquire-release, which, among other things, ensures that only the intended ordering is possible.

v2021.05-51 (26 November 2024)

Fix a vcache (aka inode) reference leak that can occur during an access rights query for a non-directory object if the neither the calling process nor the cache manager possesses tokens for the cell containing the object. The leak was introduced in v2021.05-45.
When scanning the token list for each user searching for expired tokens the user spinlock must be held to prevent races. The updated user management implementation pulled into v2021.05-45 failed to hold the spinlock.
During cache manager module shutdown reset global structures to the module initialization state. Doing so improves the likelihood that the module can be reinitialized.

v2021.05-50 (23 November 2024)

The linked list implementation of the vcache access rights cache was replaced with a btree and a last lookup cache. The btree design is more efficient for lookups and removals, and scales better as the number of uids and/or process authentication groups (pags) accessing /afs increases.
Prune access rights for expired users. Doing so reduces the cost of subsequent lookups.
Cache the access rights obtained for a uid|pag via the "anyAccess" field. Previously, a uid|pag that obtained its rights from the "anyAccess" field had to perform and fail a rights lookup on each and every vcache access.
When setting or clearing tokens for a single cell, no longer invalidate the access rights cached for the uid|pag for all cells. Instead, only invalidate the access rights cached for the cell whose tokens were set or cleared.

v2021.05-49 (16 November 2024)

The output of "tokens" command failed to report yfs-rxgk tokens was broken starting in v2021.05-46

v2021.05-48 (12 November 2024)

Preallocated buffer overflows in XDR responses (CVE-2024-10397)
The AuriStorFS and AFS3 RPC suites rely upon Sun RPC XDR to marshal binary data structures for network transfer. The AuriStor XDR implementation is derived from Sun Microsystems' Sun RPC code base. The Sun RPC XDR API permits memory for output parameters to (optionally) be preallocated which can result in various classes of memory corruption and/or memory leaks in RPC initiator processes.
The AuriStorFS v2021.05-48 release introduces additional data length validation checks within the AuriStor XDR implementation and prohibits the use of preallocated memory for string output parameters or fields. All cache managers, servers and command line tools are modified by these changes.

v2021.05-46 (28 October 2024)

Cache Manager:
- Prevent a kernel memory leak when server preferences are set via the yfs-client.conf [afsd] configuration or via "fs setserverprefs".
- Directory enumeration of a truncated directory now returns an error instead of assuming the end of the directory has been reached.
- Since AFS 3.0, the Unix cache manager has used the root identity credentials to create anonymous outgoing connections to the location service and each fileserver. However, if uid 0 is assigned a token, then those Rx connections will no longer be anonymous. Beginning with this release anonymous outgoing connections are always created with the NOPAG identity (uid 0xffffffff) instead of the root identity.
- When establishing an outgoing rxgk connection, do not fallback to the systemuser's credentials if the user's credentials resulted in a fatal error. Falling back to the systemuser's credentials can result in inappropriate use of an anonymous connection.
- Improved access rights cache correctness for YFS servers
  In prior releases, the access check logic used the file rights for any files fetched from an AuriStorFS fileserver. For files fetched from an AFS-3 fileserver (and, historically, for all files), it used the directory rights, with the (a)dmin right from the file mixed in. The (a)dmin right on a non-directory indicates that the object is owned by the authenticated user.
  This approach has some issues when combined with the access rights cache, and current fileserver callback behavior. On an AuriStorFS file server, the rights on a non-directory may be determined by the rights granted on its parent directory or, with per-file ACLs, those granted on the object itself. The fileserver will only break a non-directory's callback when a per-file ACL is changed - changing a directory ACL will not break callbacks on files within that directory. This means that changing a directory ACL will not invalidate access rights cache entries on files in tha directory, even if the effective ACL on this files has changed, and the cached rights are no longer correct.
  This release works around this by adding a new function which returns the access rights for a file hosted on an AuriStor fileserver. It uses the parent vnode information to locate the parent directory. If the parent directory isn't in the cache, or it doesn't have a valid callback, or if it has been changed since the file's access rights were cached, it clears the current access rights. Files without a parent directory must have per-file ACLs, and so their cached rights can be safely used.
  Note that files with parent vnodes may still have per-file ACLs, and that the breadcrumbing performed by the client may add parent vnode fields to vnodes which don't have them provided by the fileserver. Such vnodes may have their cached access rights cleared more frequently than necessary.
- Add a new mechanism for caching access rights within the vcache structure. This cache is protected via a vcache specific spinlock, and can be accessed without holding the GLOCK.
  This new cache mechanism returns the memory associated with cached rights back to the kernel's slab free memory pool instead of adding the unused rights structures to a cache manager managed free list. The previous cache implementation never returned allocated memory to the kernel. Instead, invalidated access rights were appended to a free access rights queue for later reuse.
- When a volume is accessed via multiple mountpoints a choice must be made regarding which mountpoint is considered to be the active (or parent) mountpoint. This release alters the behavior such that the active mountpoint is set every time a mountpoint is traversed.
  This behavior is easier to understand and is more likely to provide the expected result for a single process that repeatedly accesses volumes from multiple mountpoints. However, it can result in unexpected results when multiple processes are traversing multiple mountpoints in parallel without any synchronization.

v2021.05-45 (Not released)

v2021.05-44a (18 September 2024)

Authentication:
- AuriStorFS v2021.05-44 included an updated version of the Heimdal Kerberos framework used by AuriStorFS when acquiring yfs-rxgk and rxkad authentication tokens. The updated Heimdal included a bug which disabled the use of DNS SRV records for KDC discovery and DNS TXT records for realm discovery. As a side effect token acquisition might fail with an unable to reach any KDC in realm error. This is fixed in v2021.05-44a.

v2021.05-44 (17 August 2024)

Cache Manager:
- Since v0.192 the cache manager has failed acquire the global lock when upgrading a shared-lock to a write-lock during the execution of a background cache chunk file truncation.
Authentication:
- Neither MIT nor Heimdal gssapi nor their gss mechanisms consistently initialize the output 'minorStatus' parameter. Various functions can return either success or failure majorStatus values with minorStatus unassigned. As a result, stack garbage will be used when generating error messages. From now on libyfs_acquire will always initialize the minorStatus output variable to zero before calling into the gssapi library.
Command Parser:
- No longer accept the token "-" as a switch which eventually fails with a CMD_UNKNOWNSWITCH error. Instead, process the token as a data value.
- Optimize the processing of the loop which processes "source" command input.
- If the source command input file is "-", read from stdin.

v2021.05-41 (26 June 2024)

Rx Networking (libyfs_rx):
- A race during event creation can lead to the freeing of the event while its still in use.
- RFC1122 says that Net and Host unreachable ICMP errors might be transient and should therefore not be treated as fatal. There is no such language for the equivalent ICMPV6 errors. However, in practice ICMP6_DST_UNREACH_NOROUTE, ICMP6_DST_UNREACH_BEYONDSCOPE, and ICMP6_DST_UNREACH_ADDR can be transient.
  Linux has considered these ICMPV6 destination unreachable errors as non-fatal going back at least as far as the initial git repository commit.
  AuriStor Rx has always treated these as fatal errors which results in immediate termination of in-flight calls when received. Even if the network route corrects itself before the call timeout period expires. This release mirrors the Linux behavior and makes these errors non-fatal.
Cache Manager:
- For the first time the cache manager can detect the deletion of a volume and handle the creation of a new volume with the same name but a different volume id.
- If the location service reports the deletion of a volume, invalidate all mount points to that volume.
- RXAFS_GetCapabilities RPC failures should not be treated as a fatal error preventing failover to another replica site.
Authentication ("libyfs_acquire") used by aklog, vos, pts, bos, afsio:
- rxkad_k5 token acquisition krb5 ccache management. This release altered the krb5 credential cache management strategy once again to work around different bugs in MIT krb5 and Heimdal.
- New ACQUIRE_ERR_CRED_EXPIRED error code introduced to represent the case when a request for a service credential returns one that is already expired.
Command parser (libyfs_cmd):
- When parsing configuration files there is a depth limit of ten active inclusions. This limit was improperly enforced as a limit of ten included files instead of a depth of ten included files. As of this release it is now possible to populate an includedir directory with any number of .conf files.

v2021.05-40

Not released.

v2021.05-39 (20 May 2024)

Parallel Random Number Generation:
- AuriStorFS processes rely upon the krb5_generate_random() and RAND_bytes() functions to obtain random bytes for cryptographic operations and random counters. krb5_generate_random() internally acquires a mutex to protect internal state information. This mutex has become a significant barrier to the encryption and checksumming of Rx packets with both yfs-rxgk and rxkad.
  
  This release replaces general use of krb5_generate_random() and RAND_bytes() with a per-thread ChaCha20 CS-PRNG. This avoids the acquisition of a global mutex and permits increased parallelism on multi-core systems.
Rx Networking (libyfs_rx):
- The Rx network stack schedules a garbage collection operation to execute once per minute. This operation enforces call timeouts, destroys idle connections and destroys idle peers. The operation has historically been performed by the Rx event thread which is already responsible for performing actions in response to call RTOs, sending NAT Ping and keep-alive packets, and retrying connection challenge and reachability checks.
  
  The time complexity of the garbage collection operation is determined by the number of calls, connections, and peers. The busier the Rx endpoint the more work must be performed during each garbage collection run and the longer it takes to complete. While garbage collection is active other events cannot be processed which can interfere with the proper flow control of active calls.
  
  As with all Rx events, the garbage collection event is scheduled to execute at an absolute clock time. If the system clock drifts (or is administratively set) backwards garbage collection will not be performed until the clock catches up with the scheduled time.
  
  Another responsibility of the garbage collection procedure is to terminate calls if the system clock drifted backwards by five minutes or longer. However, when the clocked drifts backwards garbage collection is not performed until the clock has advanced beyond the point where calls require termination. As a result, calls are not terminated due to backwards clock drift and they can stall.
  
  This release re-implements the garbage collection procedure using a dedicated thread and relative waits. This change ensures that the garbage collection procedure will not prevent the execution of call related events and permits calls to be terminated when large backward clock drifts are detected.
Disk Cache Management:
- Since IBM AFS 3.5, the cache has been considered "too full" even if there exist cache files have been discarded but not yet truncated. When the cache is "too full" most operations that write to the cache will block until truncation of discarded cache files has been performed which results in unnecessary delays. This release fixes the cache such that that discarded but not yet truncated cache files do not block write operations.
- This release permits the cache truncation daemon thread to exit sooner if the cache manager is shutting down.
Improved failover when the RXGK service (co-located with each vlserver) fails to issue tokens. The failures might be the result of misconfiguration, an inability to read keys or loss of Ubik quorum.

v2021.05-38 (29 February 2024)

As with other AuriStorFS releases since the beginning of 2024, this release includes additional improvements to the Rx RPC implementation which are related to the possibility of silent data corruption when Rx jumbograms are in use. Prior releases disabled the negotiation of Rx jumbograms such that the v2021.05-37 Rx peer will refuse to send Rx jumbograms and will request that the remote peer does not send them. However, a bad actor could choose to send Rx jumbograms even though they were asked not to. v2021.05-38 introduces additional protections to ensure that a corrupt Rx jumbogram is dropped instead of being accepted.

The v2021.05-38 Rx RPC implementation also includes two optimizations. First, when Rx initiators complete a call they will no longer send an extra ACK packet to the Rx acceptor of the completed call. The sending of this unnecessary ACK creates additional work for the server which can result in increased latency for other calls being processed by the server.

Second, all AuriStor Rx services require a reach check for incoming calls from Rx peers to help protect against Distributed Reflection Denial of Service (DRDoS) attacks and execution of RPCs when the response cannot be delivered to the caller. A new reach check is required for each new call that arrives more than 60 seconds after the prior reach check completed. v2021.05-38 Rx considers the successful acknowledgment of a response DATA packet as a reach check validation. With this change reach checks will not be periodically required for a peer that completes at least one call per 60 seconds. A 1 RTT delay is therefore avoided each time a reach check can be avoided. In addition, reach checks require the service to process an additional ACK packet. Eliminating a large number of reach checks can improve overall service performance.

The final Rx RPC change in this release is specific to kernel implementations. Prior releases restricted the frequency of executing time scheduled Rx events to a granularity no smaller than 500ms. As a result an RTO timer event for a lost packet could not be shorter than 500ms even if the measured RTT for the connection is significantly smaller. The minimum RTO for a connection in AuriStor Rx is 200ms. The inability to schedule shorter timeouts impacts recovery from packet loss.

v2021.05-37 (5 February 2024)

Rx improvements:
- The v2021.05-36 release permanently disabled all use of Rx jumbograms due to a risk of silent data corruption. However, when advertising the number of acceptable datagrams in the ACK trailer a missing htonl() set the value to 16777216 instead of 1 on little-endian systems.
- When sending a PING ACK as a reachability test, ensure that the previousPacket field is properly assigned to the largest accepted DATA packet sequence number instead of zero.
- Replace the initialization state flag with two flags. One that indicates that Rx initialization began and the other that it succeeded. The first prevents multiple attempts at initialization after failure. The second prevents shutdown from accessing uninitialized structures if initialization failed.
Cache Manager Improvements:
- No longer refuse to start if both the 'cachedir' and 'memcache' options are present in the configuration file.
- New variable to store the maximum number of cache blocks used. which is accessible via /proc/fs/auristorfs/cache/blocks_used_max.

v2021.05-36 (10 January 2024)

Rx improvements:
- Permanently disable all use of Rx jumbograms due to a risk of silent data corruption.
  
  Ever since OpenAFS 1.0, and possibly before, a race condition has existed when Rx transmits packets. As the rx_call.lock is dropped when starting packet transmission, there is no protection for data that is being copied into the kernel by sendmsg(). It is critical that this packet data is not modified by another thread. However, races exist between the application, listener, and event threads that can lead to retransmissions starting whilst an original transmission is still in progress. This can lead to the packet headers being overwritten, and either the original transmission, the retransmission or both sending corrupt data to the peer.
  
  This corruption can affect the packet serial number or packet flags. It is particularly harmful when the packet flags are corrupted, as this can lead to multiple Rx packets which were intended to be sent as Rx jumbograms being delivered and misinterpreted as a single large packet. The eventual result of this depends on the Rx security class in play, but it can cause decrypt integrity errors (rxgk:crypt and rxgk:auth) or corruption of the data stream (rxnull, rxgk:clear or rxkad:auth).
  
  All AuriStorFS servers, OpenAFS 1.6 or later servers, and the Windows cache manager have been shipped with Rx jumbograms disabled by default. The UNIX cache managers however are shipped with jumbograms enabled. There are many AFS cells around the world that continue to deploy OpenAFS 1.4 or earlier fileservers which continue to negotiate the use of Rx jumbograms.
  
  It is worth noting that all AuriStorFS v0.198 and later fileservers and cache managers implement explicit checks that will recognize the corrupted application data stream and prevent corrupted file content from being stored either into an AFS vnode's backing store in the volume's object store or the cache manager's AFS cache. OpenAFS cache managers and fileservers do not have these protections.
  
  With Rx jumbograms disabled the maximum number of Rx packets in a datagram is reduced from 6 to 1; the maximum number of send and receive datagram fragments is reduced from 4 to 1; and the maximum advertised MTU is restricted to 1444 - the maximum rx packet size prior to the introduction of jumbograms in IBM AFS 3.5.
- If the rx call flow state transitions from either the RECOVERY or RESCUE states to the LOSS state as a result of an RTO resend event while writing packets to the network, cease transmission of any new DATA packets if there are packets in the resend queue.
- When the call flow state is LOSS and all packets in the resend queue have been retransmitted and yet the recovery point has not been reached, then permit new DATA packets to be sent in order to maintain a full congestion window.
- Add a safety check to prevent the estimated RTT from underflowing when the actual roundtrip time is smaller than 125us.
- Fix the computation of the padding required for rxgk encrypted packets. This bug results in packets sending 8 bytes fewer per packets than the network permits. This bug accidentally prevented the construction of Rx jumbograms when a call is protected by rxgk:crypt.
- Replace the random number generator with a more security source of random bytes.

v2021.05-33 (27 November 2023)

Rx improvements:
- Not all calls transfer enough data to be able to measure a smoothed round-trip time (SRTT). Calls which are unable to compute a SRTT should not be used to update the peer host RTO value which is used to initialize the RTO for subsequent calls.
  
  Without this change, a single DATA packet call will cause the peer host RTO to be reduced to 0ms. Subsequent calls will start with a RTO value of MAX(0, rxi_minPeerTimeout) where rxi_minPeerTimeout defaults to 200ms. If the actual measured RTO is greater than 200ms, then initial RTO will be too small resulting in premature triggering of the RTO timer and the call flow state entering the loss phase which can significantly hurt performance.
- Initialize the peer host RTO to rxi_minPeerTimeout (which defaults to 200ms) instead of one second. Although RFC6298 recommends the use of one second when no SRTT is available, Rx has long used the rxi_minPeerTimeout value for other purposes which are supposed to be consistent with initial RTO value. It should be noted that Linux TCP uses 200ms instead of one second for this purpose.
- If associating a security class with an Rx connection fails immediately place the Rx connection into an error state. A failure might occur if the security class is unable to access valid key material.
- If an incoming Rx call requires authentication and the security class is unable to successfully generate a challenge, put the incoming Rx connection into an error state and issue an abort to the caller.
- If an incoming Rx call requires authentication and the security class is able to generate a challenge but the challenge cannot be returned to Rx, then treat this as a transient error. Do not acknowledge the incoming DATA packet and do not place the Rx connection into an error state. An attempt to re-issue the challenge will be performed when the DATA packet is retransmitted.
- If an Rx call is terminated due to the expiration of the configured connection dead time, idle dead time, hard dead time, or as a result of clock drift, then send an ABORT to the peer notifying them that the call has been terminated. This is particularly important for terminated outgoing calls. If the peer does not know to terminate the call, then the call channel might be in use when the next outgoing call is issued using the same call channel. If the next incoming call is received by an in-use call channel, the receiver must drop the received DATA packet and return a BUSY packet. The call initiator will need to wait for a retransmission timeout to pass before retransmitting the DATA packet. Receipt of BUSY packets cannot be used to keep a call alive and therefore the requested call is at greater risk of timing out if the network path is congested.
aklog and krb5.log (via libyfs_acquire):
- If the linked Kerberos library implements krb5_cc_cache_match() and libacquire has been told to use an explicit principal name and credential cache, the Kerberos library might return KRB5_CC_NOTFOUND even though the requested credential cache is the correct one to use. This release will not call krb5_cc_cache_match() if the requested credential cache contains the requested principal.
Cell Service Database (cellservdb.conf):
- cellservdb.conf has been synchronized with the 31 Oct 2023 update to the grand.central.org CellServDB file.

v2021.05-32 (9 October 2023)

No significant changes for macOS compared to v2021.05-31

v2021.05-31 (25 September 2023)

New platform:
- macOS 14 Sonoma
macOS 14 Sonoma:

AuriStorFS v2021.05-29 and later installers for macOS 13 Ventura are compatible with macOS 14 Sonoma and do not need to be removed before upgrading to macOS 14 Sonoma. Installation of the macOS 14 Sonoma version of AuriStorFS is recommended.

Cache Manager:
- If an AuriStorFS cache manager is unable to use the yfs-rxgk security class when communicating with an AuriStorFS fileserver, it must assume it is IBM AFS 3.6 or OpenAFS and upgrade it to AuriStorFS if an upgrade probe returns a positive result. Once a fileserver's type is identified as AuriStorFS the type should never be reset; even if communication with the fileserver is lost or the fileserver restarts.
  
  If an AuriStorFS fileserver is replaced by an OpenAFS fileserver on the same endpoint, then the UUID of the OpenAFS must be different. As a result, the OpenAFS fileserver will be observed as distinct from the AuriStorFS fileserver that previously shared the endpoint.
  
  Prior to this release there were circumstances in which the cache manager discarded the fileserver type information and would fail to recognize the fileserver as an AuriStorFS fileserver when yfs-rxgk could not be used. This release prevents the cache manager from resetting the type information if the fileserver is marked down.
- If a fileserver's location service entry is updated with a new uniquifier value (aka version number), this indicates that one of the following might have changed:
  1. the fileserver's capabilities
  2. the fileserver's security policy
  3. the fileserver's knowledge of the cell-wide yfs-rxgk key
  4. the fileserver's endpoints
  Beginning with this release the cache manager will force the establishment of new Rx connections to the fileserver when the uniquifier changes. This ensures that the cache manager will attempt to fetch new per-fileserver yfs-rxgk tokens from the cell's RXGK service, enforce the latest security policy, and not end up in a situation where its existing tokens cannot be used to communicate with the fileserver.
aklog:
- Fix incorrect output when populating the server list for a service fails. The stashed extended error explaining the cause of the failure was not displayed.
- If a cell has neither _afs3-prserver._udp. DNS SRV records nor AFSDB records, the lookup of the cell's protection servers would fail if there is no local cell configuration details. The fallback to use _afs3-vlserver._udp. DNS SRV records did not work. This is corrected in this release.

v2021.05-30 (6 September 2023)

Do not mark a fileserver down in response to a KRB5 error code.
fs cleanacl must not store back to the file server a cleaned acl if it was inherited from a directory. Doing so will create a file acl.
Correct the generation of never expire rxkad_krb5 tokens from Kerberos v5 tickets which must have a start time of Unix epoch and an end time of 0xFFFFFFFF seconds. The incorrectly generated tokens were subject to the maximum lifetime of 30 days.
Correct the generation of the yfs-rxgk RESPONSE packet header which failed to specify the key version generation number used to encrypt the authenticator. If the actual key version is greater than zero, then the authenticator would fail to verify.
Enforce a maximum NAT ping period of 20s to ensure that NAT/PAT/firewall rules due not expire while Rx RPCs are in-flight.

v2021.05-29 (26 June 2023)

Execution of fs commands such as examine, whereis, listquota, fetchacl, cleanacl, storeacl, whoami, lsmount, bypassthreshold and getserverprefs could result in memory leaks by the AuriStorFS kernel extension.

v2021.05-27 (1 May 2023)

Fixes for bugs in vos introduced in v2021.05-26.

v2021.05-26 (17 April 2023)

Fixed a potential kernel memory leak when triggered by fs examine, fs listquota, or fs quota.
Increased logging of VBUSY, VOFFLINE, VSALVAGE, and RX_RESTARTING error responses. A log message is now generated whenever a task begins to wait as a result of one of these error responses from a fileserver. Previously, a message was only logged if the volume location information was expired or discarded.
Several changes to optimize internal volume lookups.
Faster failover to replica sites when a fileserver returns RX_RESTARTING, VNOVOL or VMOVED.
rxdebug regains the ability to report rx call flags and rx_connection flags.
The RXRPC library now terminates calls in the QUEUED state when an ABORT packet is received. This clears the call channel making it available to accept another call and reduces the work load on the worker thread pool.
Fileserver endpoint registration changes no longer result in local invalidation of callbacks from that server.
Receipt of an RXAFSCB_InitCallBackState3 RPC from a fileserver no longer resets the volume site status information for all volumes on all servers.

v2021.05-25 (28 December 2022)

The v2021.05-25 release includes further changes to RXRPC to improve reliability. The changes in this release prevent improper packet size growth. Packet size growth should never occur when a call is attempting to recover from packet loss; and is unsafe when the network path's maximum transmission unit is unknown. Packet size growth with be re-enabled in a future AuriStorFS release that includes Path MTU detection and the Extended SACK functionality.
Improved error text describing the source of invalid values in /etc/yfs/yfs-client.conf or included files and directories.

v2021.05-24 (25 October 2022)

New Platform: macOS 13 (Ventura)

v2021.05-22 (12 September 2022) and v2021.05-21 (6 September 2022)

RX RPC
- If receipt of a DATA packet causes an RX call to enter an error state, do not send the ACK of the DATA packet following the ABORT packet. Only send the ABORT packet.
- AuriStor RX has failed to count and report the number of RX BUSY packets that have been sent. Beginning with this change the sent RX BUSY packet count is once again included in the statistics retrieved via rxdebug server port -rxstats.
- Introduce minimum and maximum bounds checks on the ACK packet trailer fields. If the advertised values are out of bounds for the receiving RX stack, do not abort the call but adjust the values to be consistent with the local RX RPC implementation limits. These changes are necessary to handle broken RX RPC implementations or prevent manipulation by attackers.

New to v2021.05-20 (15 August 2022) and v2021.05-19 (13 August 2022)

RX RPC
- Include the DATA packet serial number in the transmitted reachability check PING ACK. This permits the reachability test ACK to be used for RTT measurement.
- Do not terminate a call due to an idle dead timeout if there is data pending in the receive queue when the timeout period expires. Instead deliver the received data to the application. This change prevents idle dead timeouts on slow lossy network paths.
- Fix assignment of RX DATA, CHALLENGE, and RESPONSE packet serial numbers in macOS (KERNEL). Due to a mistake in the implementation of atomic_add_and_read the wrong serial numbers were assigned to outgoing packets.

New to v2021.05-18 (12 July 2022)

Cache Manager
- Prevent a kernel memory leak of less than 64 bytes for each bulkstat RPC issued to a fileserver. Bulkstat RPCs can be frequently issued and over time this small leak can consume a large amount of kernel memory. Leak introduced in AuriStorFS v0.196.
- The Perl::AFS module directly executes pioctls via the OpenAFS compatibility pioctl interface instead of the AuriStorFS pioctl interface. When Perl::AFS is used to store an access control list (ACL), the deprecated RXAFS_StoreACL RPC would be used in place of the newer RXAFS_StoreACL2 or RXYFS_StoreOpaqueACL2 RPCs. This release alters the behavior of the cache manager to use the newer RPCs if available on the fileserver and fallback to the deprecated RPC. The use of the deprecated RPC was restricted to use of the OpenAFS pioctl interface.
RX RPC
- Handle a race during RX connection pool probes that could have resulted in the wrong RX Service ID being returned for a contacted service. Failure to identify that correct service id can result in a degradation of service.
- The Path MTU detection logic sends padded PING ACK packets and requests a PING_RESPONSE ACK be sent if received. This permits the sender of the PING to probe the maximum transmission unit of the path. Under some circumstances attempts were made to send negative padding which resulted in a failure when sending the PING ACK. As a result, the Path MTU could not be measured. This release prevents the use of negative padding.
Preparation for supporting macOS 13 Ventura when it is released in Fall 2022.
Some shells append a slash to an expanded directory name in response to tab completion. These trailing slashes interfered with "fs lsmount", "fs flushmount" and "fs removeacl" processing. This release includes a change to prevent these commands from breaking when presented a trailing slash.

New to v2021.05-17 (16 May 2022)

Cell Service Database Updates
- Update cern.ch, ics.muni.cz, ifh.de, cs.cmu.edu, qatar.cmu.edu, it.kth.se
- Remove uni-hohenheim.de, rz-uni-jena.de, mathematik.uni-stuttgart.de, stud.mathematik.uni-stuttgart.de, wam.umd.edu
- Add ee.cooper.edu
- Restore ams.cern.ch, md.kth.se, italia
Fix parsing of [afsd] rxwindow configuration which can be used to specified a non-default send/receive RX window size. The current default is 128 packets.
RX Updates
- Add nPacketsReflected

Logging improvements

Cache directory validation errors log messages now include the cache directory path.

Log the active configuration path if "debug" logging is enabled.

More details of rxgk token extraction failures.

New to v2021.05-16 (24 March 2022)

RX - Previous releases re-armed the Retransmission Timeout (RTO) each time a new unacknowledged packet is acknowledged instead of when a new leading edge packet is acknowledged. If leading edge data packet and its retransmission are lost, the call can remain in the "recovery" state where it continues to send new data packets until one of the following is true:
. the maximum window size is reached
. the number of lost and resent packets equals 'cwind'
at which point there is nothing left to transmit. The leading edge data packet can only be retransmitted when entering the "loss" state but since the RTO is reset with each acknowledged packet the call stalls for one RTO period after the last transmitted data packet is acknowledged.

This poor behavior is less noticiable with small window sizes and short lived calls. However, as window sizes and round-trip times increase the impact of a twice lost packet becomes significant.

RX - Never set the high-order bit of the Connection Epoch field. RX peers starting with IBM AFS 3.1b through AuriStor RX v0.191 ignore the source endpoint when matching incoming packets to RX connections if the high-order epoch bit is set. Ignoring the source endpoint is problematic because it can result in a call entering a zombie state whereby all PING ACK packets are immediately responded to the source endpoint of the PING ACK but any delayed ACK or DATA packets are sent to the endpoint bound to the RX connection. An RX client that moves from one network to another or which has a NAT|PAT device between it and the service can find themselves stuck.

Starting with AuriStor RX v0.192 the high-order bit is ignored by AuriStor RX peer when receiving packets. This change to always clear the bit prevents IBM AFS and OpenAFS peers from ignoring the source endpoint.

RX - The initial packetSize calculation for a call is altered to require that all constructed packets before the receipt of the first ACK packet are eligible for use in jumbograms if and only if the local RX stack has jumbograms enabled and the maximum MTU is large enough. By default jumbograms are disabled for all AuriStorFS services. This change will have a beneficial impact if jumbograms are enabled via configuration; or when testing RX performance with "rxperf".

New fs whereis -noresolve option displays the fileservers by network endpoint instead of DNS PTR record hostname.

New to v2021.05-15 (24 January 2022)

kernel - fixed YFS_RXGK service rx connection pool leak

New to v2021.05-14 (20 January 2022)

fs mkmount permit mount point target strings longer than 63 characters.
afsd enhance logging of yfs-rxgk token renewal errors.

afsd gains a "principal = configuration option for use with keytab acquisition of yfs-rxgk tokens for the cache manager identity.

kernel - Avoid unnecessary rx connection replacement by racing threads after token replacement or expiration.

kernel - Fix a regression introduced in v2021.05 where an anonymous combined identity yfs-rxgk token would be replaced after three minutes resulting in the connection switching from yfs-rxgk to rxnull.

kernel - Fix a regression introduced in v0.208 which prevented the invalidation of cached access rights in response to a fileserver callback rpc. The cache would be updated after the first FetchStatus rpc after invalidation.

kernel - Reset combined identity yfs-rxgk tokens when the system token is replaced.

kernel - The replacement of rx connection bundles in the cache manager to permit more than four simultaneous rx calls per uid/pag with trunked rx connections introduced the following regressions in v2021.05.

a memory leak of discarded rx connection objects

failure of NAT ping probes after replacement of an connection

inappropriate use of rx connections after a service upgrade failure

All of these regressions are fixed in patch 14.

New to v2021.05-12 (7 October 2021)

fs ignorelist -type afsmountdir in prior releases could prevent access to /afs.

Location server rpc timeout restored to two minutes instead of twenty minutes.

Location server reachability probe timeout restored to six seconds instead of fifty seconds.

Cell location server upcall results are now cached for fifteen seconds.

Multiple kernel threads waiting for updated cell location server reachability probes now share the results of a single probe.

RX RPC implementation lock hierarchy modified to prevent a lock inversion.

RX RPC client connection reference count leak fixed.

RX RPC deadlock during failed connection service upgrade attempt fixed..

New to v2021.05-9 (25 October 2021)

First public release for macOS 12 Monterey build using XCode 13. When upgrading macOS to Monterey from earlier macOS releases, please upgrade AuriStorFS to v2021.05-9 on the starting macOS release, upgrade to Monterey and then install the Monterey specific v2021.05-9 release.

Improved logging of "afsd" shutdown when "debug" mode is enabled.

Minor RX network stack improvements

New to v2021.05-3 (10 June 2021)

Fix for [cells] cellname = {...} without server list.

New to v2021.05 (31 May 2021)

Multi-homed location servers are finally managed as a single server instead of treating each endpoint as a separate server. The new functionality is a part of the wholesale replacement of the former cell management infrastructure. Location server communication is now entirely managed as a cluster of multi-homed servers for each cell. The new infrastructure does not rely upon the global lock for thread safety.

This release introduces a new infrastructure for managing user/pag entities and tracking their per cell tokens and related connection pools.

Expired tokens are no longer immediately deleted so that its possible for them to be listed by "tokens" for up to two hours.

Prevent a lock inversion introduced in v0.208 that can result in a deadlock involving the GLOCK and the rx call.lock. The deadlock can occur if a cell's list of location servers expires and during the rebuild an rx abort is issued.

Add support for rxkad "auth" mode rx connections in addition to "clear" and "crypt". "auth" mode provides integrity protection without privacy.

Add support for yfs-rxgk "clear" and "auth" rx connection modes.

Do not leak a directory buffer page reference when populating a directory page fails.

Re-initialize state when populating a disk cache entry using the fast path fails and a retry is performed using the slow path. If the data version changes between the attempts it is possible for truncated disk cache data to be treated as valid.

Log warnings if a directory lookup operation fails with an EIO error. An EIO error indicates that an invalid directory header, page header, or directory entry was found.

Do not overwrite RX errors with local errors during Direct-I/O and StoreMini operations. Doing so can result in loss of VBUSY, VOFFLINE, UAENOSPC, and similar errors.

Correct a direct i/o code path which could overwrite a fileserver returned error code with a local RXGEN_CC_UNMARSHAL error.

Correct the StoreMini code path which could overwrite a fileserver returned error code with a local RXGEN_CC_UNMARSHAL error.

Ensure the rx call object is not locked when writing to the network socket.

Removed all knowledge of the KERNEL global lock from RX. Acquiring the GLOCK from RX is never safe if any other lock is held. Doing so is a lock order violation that can result in deadlocks.

Fixed a race in the opr_reservation system that could produce a cache entry reference undercount.

If a directory hash chain contains a circular link, a buffer page reference could be leaked for each traversal.

Each AFS3 directory header and page header contains a magic tag value that can be used in a consistency check but was not previously checked before use of each header. If the header memory is zero filled during a lookup, the search would fail producing an ENOENT error. Starting with this release the magic tag values are validated on each use. An EIO error is returned if there is a tag mismatch.

"fs setcrypt -crypt auth" is now a permitted value. The "auth" mode provides integrity protection but no privacy protection.

Add new "aklog -levels " option which permits requesting "clear" and "auth" modes for use with yfs-rxgk.

Update MKShim to Apple OpenSource MITKerberosShim-79.

Report KLL errors via a notification instead of throwing an exception which (if not caught) will result in process termination.

If an exception occurs while executing "unlog" catch it and ignore it. Otherwise, the process will terminate.

New to v2021.04 (22 April 2021)

Primarily bug fixes for issues that have been present for years.

A possibility of an infinite kernel loop if a rare file write / truncate pattern occurs.

A bug in silly rename handling that can prevent cache manager initiated garbage collection of vnodes.

New to v0.209 (13 March 2021)

fs setserverprefs and fs getserverprefs updated to support IPv6 and CIDR specifications.

Improved error handling during fetch data and store data operations.

Prevents a race between two vfs operations on the same directory which can result in caching of out of date directory contents.

Use cached mount point target information instead of evaluating the mount point's target upon each access.

Avoid rare data cache thrashing condition.

Prevent infinite loop if a disk cache error occurs after the first page in a chunk is written.

Network errors are supposed to be returned to userspace as ETIMEDOUT. Previously some were returned as EIO.

When authentication tokens expire, reissue the fileserver request anonymously. If the anonymous user does not have permission either EACCES or EPERM will be returned as the error to userspace. Previously the vfs request would fail with an RXKADEXPIRED or RXGKEXPIRED error.

If growth of an existing connection vector fails, wait on a call slot in a previously created connection instead of failing the vfs request.

Volume and fileserver location query infrastructure has been replaced with a new modern implementation.

Replace the cache manager's token management infrastructure with a new modern implementation.

New to v0.206 (12 January 2021) - Bug fixes

Prevents a possible panic during unmount of /afs.

Improved failover and retry logic for offline volumes.

New to v0.205 (24 December 2020) - Bug fixes

Volume name-to-id cache improvements

Fix expiration of name-to-id cache entries

Control volume name-to-id via sysctl

Query volume name-to-id statistics via sysctl

Improve error handling for offline volumes

Fix installer to prevent unnecessary installation of Rosetta 2 on Apple Silicon

New to v0.204 (25 November 2020) - Bug fix for macOS Big Sur

v0.204 prevents a kernel panic on Big Sur when AuriStorFS is stopped and restarted without an operating system reboot.

introduces a volume name-to-id cache independent of the volume location cache.

New to v0.203 (13 November 2020) - Bug fix for macOS

v0.203 prevents a potential kernel panic due to network error.

New to v0.201 (12 November 2020) - Universal Big Sur (11.0) release for Apple Silicon and Intel

v0.201 introduces a new cache manager architecture on all macOS versions except for High Sierra (10.12). The new architecture includes a redesign of:

kernel extension load

kernel extension unload (not available on Big Sur)

/afs mount

/afs unmount

userspace networking

The conversion to userspace networking will have two user visible impacts for end users:
The Apple Firewall as configured by System Preferences -> Security & Privacy -> Firewall is now enforced. The "Automatically allow downloaded signed software to receive incoming connections" includes AuriStorFS.

Observed network throughput is likely to vary compared to previous releases.

On Catalina the "Legacy Kernel Extension" warnings that were displayed after boot with previous releases of AuriStorFS are no longer presented with v0.201.

AuriStorFS /afs access is expected to continue to function when upgrading from Mojave or Catalina to Big Sur. However, as AuriStorFS is built specifically for each macOS release, it is recommended that end users install a Big Sur specific AuriStorFS package.
AuriStorFS on Apple Silicon supports hardware accelerated aes256-cts-hmac-sha1-96 and aes128-cts-hmac-sha1-96 using AuriStor's proprietary implementation.

New to v0.200 (4 November 2020) - Final release for macOS El Capitan (10.11)

The network path between a client and a server often traverses one or more network segments separated by NAT/PAT devices. If a NAT/PAT times out a RPCs endpoint translation mid-call, this can result in an extended delay before failure and the server being marked down, or worse, a call that never terminates and a client that appears to hang until the fileserver is restarted.

This release includes significant changes to the RX stack and the UNIX cache manager to detect such conditions, fail the calls quickly and detect when it is safe to retry the RPC.

NAT/PAT devices that drop endpoint mappings while in use are anti-social and can result in unwanted delays and even data loss. Whenever possible they should be avoided when they can be. That said, the changes in this release are a huge step to making the loss of endpoint mappings tolerable.

Fix segmentation fault of Backgrounder when krb5_get_credentials() fails due to lack of network connectivity.

Fix the "afsd" rxbind option which was ignored if the default port, 7001, is in use by another process on the system.

If a direct i/o StoreData or FetchData RPC failed such that it must be retried, the retried RPC would fail due to an attempt to Fetch or Store the wrong amount of data. This is fixed.

Servers are no longer marked down if RPCs fail with RX_CALL_PEER_RESET, RX_CALL_EXCEEDS_WINDOW, or RX_PROTOCOL_ERROR. RPCs that are safe to retry are retried.

Fixed a race between a call entering error state and call completion that can result in a the call remaining in the DALLY state and the connection channel remaining in use. If this occurs during process or system shutdown it can result in a deadlock.

During shutdown cancel any pending delayed aborts to prevent a potential deadlock. If a deadlock occurs when unloading a kernel module a reboot will be required.

Updated cellservdb.conf

New to v0.197.1 (31 August 2020) and v0.198 (10 October 2020)

Prevent Dead vnode has core/unlinkedel/flock panic introduced in v0.197.

New to v0.197 (26 August 2020)

A new callback management framework for UNIX cache managers reduces the expense of processing volume callback RPCs from O(number of vcache objects) to O(1). A significant amount of lock contention has been avoided. The new design reduces the risk of the single callback service worker thread blocking. Delays in processing callbacks on a client can adversely impact fileserver performance and other clients in the cell.

Bulk fetch status RPCs are available on macOS for the first time. Bulk fetch status permits optimistic caching of vnode status information without additional round-trips. Individual fetch status RPCs are no longer issued if a bulk status fails to obtain the required status information.

Hardware accelerated crypto is now available for macOS cache managers. AuriStor's proprietary aes256-cts-hmac-sha1-96 and aes256-cts-hmac-sha512-384 implementations leverage Intel processor extensions: AESNI AVX2 AVX SSE41 SSSE3 to achieve the fastest encrypt, decrypt, sign and verify times for RX packets.

This release optimizes the removal of "._" files that are used to store extended attributes by avoiding unnecessary status fetches when the directory entry is going to be removed.

When removing the final directory entry for an in-use vnode, the directory entry must be silly renamed on the fileserver to prevent removal of the backing vnode. The prior implementation risked blindly renaming over an existing silly rename directory entry.

Behavior change! When the vfs performs a lookup on ".", immediately return the current vnode.

if the object is a mount point, do not perform fakestat and attempt to resolve the target volume root vnode.

do not perform any additional access checks on the vnode. If the caller already knows the vnode the access checks were performed earlier. If the access rights have changed, they will be enforced when the vnode is used just as they would have if the lookup of "." was performed within the vfs.

do not perform a fetch status or fetch data rpcs. Again, the same as if the lookup of "." was performed within the vfs.

Volumes mounted at more than one location in the /afs namespace are problematic on more than one operating system that do not expect directories to have more than one parent. It is particularly problematic if a volume is mounted within itself. Starting with this release any attempt to traverse a mountpoint to the volume containing the mountpoint will fail with ENODEV.

When evaluating volume root vnodes, ensure that the vnode's parent is set to the parent directory of the traversed mountpoint and not the mountpoint. Vnodes without a parent can cause spurious ENOENT errors on Mojave and later.

v0.196 was not publicly released.

New to v0.195 (14 May 2020)

This is a CRITICAL update for AuriStorFS macOS clients.

In Sep 2019 AuriStorFS v0.189 was released which provided faster and less CPU intensive writing of (>64GB) large files to /afs. These improvements introduced a hash collision bug in the store data path of the UNIX cache manager which can result in file corruption. If a hash collision occurs between two or more files that are actively being written to via cached I/O (not direct I/O), dirty data can be discarded from the auristorfs cache before it is written to the fileserver creating a file with a range of zeros (a hole) on the fileserver. This hole might not be visible to the application that wrote the data because the lost data was cached by the operating system. This bug has been fixed in v0.195 and it is for this reason that v0.195 has been designated a CRITICAL release for UNIX/Linux clients.

While debugging a Linux SIGBUS issue, it was observed that receipt of an ICMP network error in response to a transmitted packet could result in termination of an unrelated rx call and could mark a server down. If the terminated call is a StoreData RPC, permanent data loss will occur. All Linux clients derived from the IBM AFS code base experience this bug. The v0.195 release prevents this behavior.

This release includes changes that impact all supported UNIX/Linux cache managers. On macOS there is reduced lock contention between kernel threads when the vcache limit has been reached.

The directory name lookup cache (DNLC) implementation was replaced. The new implementation avoids the use of vcache pointers which did not have associated reference counts, and eliminates the invalidation overhead during callback processing. The DNLC now supports arbitrary directory name lengths; the prior implementation only cached entries with names not exceeding 31 characters.

Prevent matching arbitrary cell name prefixes as aliases. For example "/afs/y" should not be an alias for "your-file-system.com". Some shells, for example "zsh", query the filesystem for names as users type. Delays between typed characters result in filesystem lookups. When this occurs in the /afs dynroot directory, this could result in cellname prefix string matches and the dynamic creation of directory entries for those prefixes.

New to v0.194 (2 April 2020)

This is a CRITICAL release for all macOS users. All prior macOS clients whether AuriStorFS or OpenAFS included a bug that could result in data corruption either when reading or writing.

This release also fixes these other issues:

sign and notarize installer plugin "afscell" bundle. The lack of digital signature prevented the installer from prompting for a cellname on some macOS versions.

prevent potential for corruption when caching locally modified directories.

v0.193 was withdrawn due to a newly introduced bug that could result in data corruption.

New to v0.192 (30 January 2020)

The changes improve stability, efficiency, and scalability. Post-0.189 changes exposed race conditions and reference count errors which can lead to a system panic or deadlock. In addition to addressing these deficiencies this release removes bottlenecks that restricted the number of simultaneous vfs operations that could be processed by the AuriStorFS cache manager. The changes in this release have been successfully tested with greater than 400 simultaneous requests sustained for for several days.

New to v0.191 (16 December 2019)

Restore keyed cache manager capability broken in v0.189.

Add kernel module version string to AuriStorFS Preference Pane.

Other kernel module bug fixes.

New to v0.190 (14 November 2019)

Short-circuit busy volume retries after volume or volume location entry is removed.

New to v0.189 (28 October 2019)

Faster "git status" operation on repositories stored in /afs.

Faster and less CPU intensive writing of (>64GB) large files to /afs. Prior to this release writing files larger than 1TB might not complete. With this release store data throughput is consistent regardless of file size. (See "UNIX Cache Manager large file performance improvements" later in this file).

macOS Catalina (8 October 2019)

AuriStorFS v0.188 released for macOS Catalina (10.15)

New to v0.188 (23 June 2019)

Increased clock resolution for timed waits from 1s to 1ns

Added error handling for rx multi rpcs interrupted by signals

New to v0.186 (29 May 2019)

v0.184 moved the /etc/yfs/cmstate.dat file to /var/yfs. With this change afsd would fail to start if /etc/yfs/cmstate.dat exists but contains invalid state information. This is fixed.

v0.184 introduced a potential deadlock during directory processing. This is fixed.

Handle common error table errors obtained outside an afs_Analyze loop. Map VL errors to ENODEV and RX, RXKAD, RXGK errors to ETIMEDOUT

Log all server down and server up events. Transition events from server probes failed to log messages.

RX RPC networking:

If the RPC initiator successfully completes a call without consuming all of the response data fail the call by sending an RX_PROTOCOL_ERROR ABORT to the acceptor and returning a new error, RX_CALL_PREMATURE_END, to the initiator.
Prior to this change failure to consume all of the response data would be silently ignored by the initiator and the acceptor might resend the unconsumed data until any idle timeout expired. The default idle timeout is 60 seconds.

Avoid transmitting ABORT, CHALLENGE, and RESPONSE packets with an uninitialized sequence number. The sequence number is ignored for these packets but set it to zero.

New to v0.184 (26 March 2019)

The initial congestion window has been reduced from 10 Rx packets to 4. Packet reordering and loss has been observed when sending 10 Rx packets via sendmmsg() in a single burst. The lack of udp packet pacing can also increase the likelihood of transmission stalls due to ack clock variation.

The UNIX Cache Manager underwent major revisions to improve the end user experience by revealing more error codes, improving directory cache efficiency, and overall resiliency. The cache manager implementation was redesigned to be more compatible with operating systems such as Linux and macOS that support restartable system calls. With these changes errors such as "Operation not permitted", "No space left on device", "Quota exceeded", and "Interrupted system call" can be reliably reported to applications. Previously such errors might have been converted to "I/O error".

New to v0.180 (9 November 2018)

RX reliability and performance improvements for high latency and/or lossy network paths such as public wide area networks.

A fix for a macOS firewall triggered kernel panic introduced in v0.177.

New to v0.177 (17 October 2018)

A fix to AuriStor's RX implementation bug introduced in v0.176 that interferes with communication with OpenAFS and IBM Location and File Services.

New to v0.176 (3 October 2018)

AuriStor's RX implementation has undergone a major upgrade of its flow control model. Prior implementations were based on TCP Reno Congestion Control as documented in RFC5681; and SACK behavior that was loosely modelled on RFC2018. The new RX state machine implements SACK based loss recovery as documented in RFC6675, with elements of New Reno from RFC5682 on top of TCP-style congestion control elements as documented in RFC5681. The new RX also implements RFC2861 style congestion window validation.

When sending data the RX peer implementing these changes will be more likely to sustain the maximum available throughput while at the same time improving fairness towards competing network data flows. The improved estimation of available pipe capacity permits an increase in the default maximum window size from 60 packets (84.6 KB) to 128 packets (180.5 KB). The larger window size increases the per call theoretical maximum throughput on a 1ms RTT link from 693 mbit/sec to 1478 mbit/sec and on a 30ms RTT link from 23.1 mbit/sec to 49.39 mbit/sec.

Improve shutdown performance by refusing to give up callbacks to known unreachable file servers and apply a shorter timeout period for the rest.

Permit RXAFSCB_WhoAreYou to be successfully executed after an IBM AFS or OpenAFS fileserver unintentionally requests an RX service upgrade from RXAFSCB to RXYFSCB.

RXAFS timestamps are conveyed in unsigned 32-bit integers with a valid range of 1 Jan 1970 (Unix Epoch) through 07 Feb 2106. UNIX kernel timestamps are stored in 32-bit signed integers with a valid range of 13 Dec 1901 through 19 Jan 2038. This discrepency causes RXAFS timestamps within the 2038-2106 range to display as pre-Epoch.

RX Connection lifecycle management was susceptible to a number of race conditions that could result in assertion failures, the lack of a NAT ping connection to each file server, and the potential reuse of RX connections that should have been discarded.

This release includes a redesigned lifecycle that is thread safe, avoids assertions, prevents NAT ping connection loss, and ensures that discarded connections are not reused.

The 0.174 release unintentionally altered the data structure returned to xstat_cm queries. This release restores the correct wire format.

Since v0.171, if a FetchData RPC fails with a VBUSY error and there is only one reachable fileserver hosting the volume, then the VFS request will immediately with an ETIMEDOUT error ("Connection timed out").

v0.176 corrects three bugs that contributed to this failure condition. One was introduced in v0.171, another in 0.162 and the final one dates to IBM AFS 3.5p1.

The intended behavior is that a cache manager, when all volume sites fail an RPC with a VBUSY error, will sleep for up to 15 seconds and then retry the RPC as if the VBUSY error had never been received. If the RPC continues to receive VBUSY errors from all sites after 100 cycles, the request will be failed with EWOULDBLOCK ("Operation would block") and not ETIMEDOUT.

Prefer VOLMISSING and VOLBUSY error states to network error states when generating error codes to return to the VFS layer. This will result in ENODEV ("No such device") errors when all volume sites return VNOVOL or VOFFLINE errors and EWOULDBLOCK ("Operation would block") errors when all volume sites return VBUSY errors. (v0.176)

New to v0.174 (24 September 2018)

macOS Mojave (10.14) support

New to v0.170 (27 April 2018)

Faster processing of cell configuration information by caching service name to port information.

RX call sequence number rollover to permit calls that require the transmission of more than 5.5TB of data.

Command parser Daylight Saving Time bug fix

Fix a bug that prevented immediate access to a mount point created with "fs mkmount" on the same machine.

Fix the setting of "[afsd] sysnames = " during cache manager startup.

New to v0.168 (6 March 2018)

Corrects "fs setacl -negative" processing [CVE-2018-7168]

Improved reliability for keyed cache managers. More persistent key acquisition renewals.

Major refresh to cellservdb.conf contents.

DNS SRV and DNS AFSDB records now take precedence when use_dns = yes

Kerberos realm hinting provided by
kerberos_realm = [REALM]

DNS host names are resolved instead of reliance on hard coded IP addresses
The cache manager now defaults to sparse dynamic root behavior. Only thiscell and those cells that are assigned aliases are included in /afs directory enumeration at startup. Other cells will be dynamically added upon first access.
Several other quality control improvements.

New to v0.167 (7 December 2017)

Addresses a critical remote denial of service vulnerability [CVE-2017-17432]
Alters the volume location information expiration policy to reduce the risk of single points of failures after volume release operations.
'fs setquota' when issued with quota values larger than 2TB will fail against OpenAFS and IBM AFS file servers
Memory management improvements for the memory caches.

New to v0.164 (11 November 2017)

Internal cache manager redesign. No new functionality.

New to v0.160 (21 September 2017)

Support for OSX High Sierra's new Apple File System (APFS). Customers must upgrade to v0.160 or later before upgrading to OSX High Sierra.
Reduced memory requirements for rx listener thread
Avoid triggering a system panic if an AFS local disk cache file is deleted or becomes inaccessible.
Fixes to "fs" command line output

New to v0.159 (7 August 2017)

Improved failover behavior during volume maintenance operations
Corrected a race that could lead the rx listener thread to enter an infinite loop and cease processing incoming packets.

New to v0.157 (12 July 2017)

Bundled with Heimdal 7.4 to address CVE-2017-11103 (Orpheus' Lyre puts Kerberos to sleep!)
"vos" support for volume quotas larger than 2TB.
"fs flushvolume" works
Fixed a bug that can result in a system panic during server capability testing

New to v0.150

AuriStorFS file server detection improvements

New to v0.149

rxkad encryption is enabled by default. Use "fs setcrypt off" to disable encryption when tokens are available.
Fix a bug in atomic operations on Sierra and El Capitan which could adversely impact Rx behavior.

New to v0.128

Extended attribute ._ files are automatically removed when the associated files are unlinked
Throughput improvements when sending data

New to v0.121

OSX Sierra support

New to v0.117

Cache file moved to a persistent location on local disk
AuriStor File System graphics
Improvements in Background token fetch functionality
Fixed a bug introduced in v0.44 that could result in an operating system crash when enumerating AFS directories containing Unicode file names (v0.106)
El Capitan security changes prevented Finder from deleting files and directories. As of v0.106, the AuriStor OSX client implements the required functionality to permit the DesktopHelperService to securely access the AFS cache as the user permitting Finder to delete files and directories.

Features:

Not vulnerable to OPENAFS-SA-2015-007.
Office 2011 can save to /afs.
Office 2016 can now save files to /afs.
OSX Finder and Preview can open executable documents without triggering a "Corrupted File" warning. .AI, .PDF, .TIFF, .JPG, .DOCX, .XLSX, .PPTX, and other structured documents that might contain scripts were impacted.
All file names are now stored to the file server using Unicode UTF-8 Normalization Form C which is compatible with Microsoft Windows.
All file names are converted to Unicode UTF-8 Normalization Form D for processing by OSX applications.

Known issues:

None

macOS Installer (13.0 Ventura)

Release Notes

Known Issues

If the Kerberos default realm is not configured, a delay of 6m 59s can occur before the AuriStorFS Backgrounder will acquire tokens and display its icon in the macOS menu. This is the result of macOS performing a Bonjour (MDNS) query in an attempt to discover the local realm.
"aklog" will fail to acquire tokens if the current working directory is located in an inaccessible AFS volume. When the "cwd" is inaccessible, the Heimdal framework is unable to initialize the Kerberos v5 context and reports an out of memory error.

New v2021.05-64 (22 Aug 2025)

The v2021.05-64 release fixes a race condition which can lead to a kernel panic when executing the "unlog" command.

v2021.05-63 (31 May 2025)

When AuriStorFS cache managers added support for per-file ACLs in v0.1, the masking of access rights based upon the inode's S_IRWXU mode bits were skipped when reading objects from AuriStorFS fileservers. The masking continued to be applied when reading objects from OpenAFS fileservers. This release applies S_IRWXU mode bits masking of granted access rights to both AuriStorFS and OpenAFS fileservers.
The existing fake stat mountpoint evaluation logic assumes that the fileserver returned a AFSFetchStatus.ParentVnode that refers to the parent directory from which the mountpoint was located. However, this is only guaranteed to be true if the mountpoint does not have a per-file ACL (which is permitted) and has never been cross-directory hard linked (which is also permitted).
This release improves the error handling logic to fail the fake stat evaluation if the AFSFetchStatus.ParentVnode does not match expectations.
This release fixes a metadata inconsistency when a mountpoint is created when disconnected mode is enabled. A mountpoint should be reported to the vfs as a directory and not a symlink.
The cache manager must track for each inode whether the object's status infomration was received from an AFS3 fileserver or an AuriStorFS fileserver. This release alters when that information is recorded to ensure that it is recorded each time the inode's status information is updated by a fileserver response.
This knowledge is required to properly determine how to interpret the status information.
Directory entries referring to mountpoints and volume roots must not be stored within the Directory Name Lookup Cache (DNLC). Instead, full lookups must be performed each time to ensure proper mountpoint resolution.
Prior to this release locally created mountpoints were stored into the DNLC by mistake.

v2021.05-62 (24 April 2025)

macOS Sequoia v15.4 vs Cisco AnyConnect VPN

Beginning with macOS 15.4, use of Cisco AnyConnect VPN breaks DNS SRV, AFSDB and TXT record lookups performed using the cross-platform DNS resolver API. As a result, clients which rely upon DNS-based zero-configuration resolution of Kerberos realms and/or AFS cells database services are unable to access those resources. This release switches the macOS builds to use the Apple DNS API.
The lookup failures impacted all command line tools, clients and services.
Note: Communication failures can occur if the VPN is configured by the organization to block the transmission of fragmented UDP packets. Set [afsd] rxmaxmtu = 1328 to ensure all RX packets are small enough to traverse the VPN link.
This release improves consistency of vnode metadata interpretation when the status information is received via an RPC which does not return a callback.

v2021.05-61 (31 March 2025)

Instead of allocating the same sized hash table for the vcache and dcache objects regardless of how many elements are expected to be stored, scale the hash table sizes. The vcache hash table will scale from 4096 buckets to 131072 buckets with a target of 4 entries per bucket. The dcache hash tables will scale from 4096 buckets to 65536 buckets with a target of 2 entries per bucket.
Historically, afs cache managers which create symlinks (including mountpoints) locally allocate a fake infinite lifetime callback to permit applications to use the symlink (or mountpoint) without requiring FetchStatus RPCs to be issued to the fileserver. This approach has drawbacks because it means that fileservers will not issue a callback break when the volume containing the symlink is taken offline, moved, deleted, etc. It also means that a callback will not be sent if another cache manager removes the symlink.
Beginning with this release the AuriStorFS cache manager will no longer allocate a fake infinite lifetime callback and will instead rely upon fileserver allocated callbacks.
"fs makemount" is supposed to create a mountpoint even if the target volume does not exist or is inaccessible. v2021.05 introduced support for creating mountpoints which kafs which broke the ability to create mountpoints to non-existent volumes.

v2021.05-60 (6 March 2025)

The cache manager retries fileserver RPCs up to 100 times when the fileserver replies with VBUSY, VOFFLINE or VSALVAGE. The cache manager fails the application request after the limit is reached. Prior to this release, the failure would result in the cache manager returning an ENODEV error to the application. But ENODEV indicates that the required volume does not exist and the fileserver has not indicated that the volume does not exist. The cache manager will now return EWOULDBLOCK to the application because the volume might be temporarily inaccessible.
Many OpenAFS and AuriStorFS fileservers can return VNOVOL as a side effect of a race between a volume going offline so that control can be given to the Volume Management Service (volserver) and an incoming RPC requesting that the volume be attached. A VNOVOL can also be returned because the volume was temporarily deleted from the vice partition prior to being replaced by a new clone, or prior to the fileserver marking the volume group as having been moved to another fileserver.
The fileserver replying to an RPC with VNOVOL cannot be trusted to mean that the volume has been permanently removed unless it is matched by the Location Service reporting the Volume Location (VL) entry has been deleted. From now on when a VNOVOL response is received and the VL entry exists, the cache manager will return EWOULDBLOCK instead of ENODEV to the application.
Prior to this release, any VL error code returned from the Location Service would be interpreted as meaning that the requested VL entry does not exist. Starting with this release only the VL_NOENT error will be interpreted as the VL entry does not exist.
Restrict the internal use of the ENOENT error code so the only use of ENOENT is to indicate that a directory entry does not exist. Previously, ENOENT could be returned if a disk cache entry could not be allocated or found. A disk cache related ENOENT returned when processing a directory might have been confused for the requested directory entry not existing.
Restrict the internal use of the ENODEV error code so that the only meaning of ENODEV is that the required volume does not exist. Previously, ENODEV could be returned due to transient errors when failing to contact the Location Service.

v2021.05-59 (13 February 2025)

The fix for garbage collection of idle unixuser objects introduced in v2021.05-56 was incomplete. As a result, Rx client connections might not be garbage collected.

v2021.05-57 (9 February 2025)

No new changes for macOS.

v2021.05-56 (7 February 2025)

Minor bug fixes to the Rx RPC implementation
The fix for garbage collection of idle unixuser objects introduced in v2021.05-55 was incomplete. As a result, Rx client connections might not be garbage collected.

v2021.05-55 (23 January 2025)

Minor bug fixes to the Rx RPC implementation
During garbage collection of idle unixuser objects, any Rx client connections associated with the unixuser (uid or pag) that were in use by another thread would not have their cache manager reference dropped. As a result the Rx client connections could never be garbage collected which is problematic on systems which churn through many unixusers. The behavior has been changed so that unixusers objects with in use Rx connections will not be freed. This permits future attempts to garbage collect unixusers to do so only after the Rx connections can also be freed.
Although this misbehavior has been present for decades, it is only since v2021.05-52 that the problem has been observed. It is possible that the changes to reduce the holding of the GLOCK has created a window of opportunity for connections to be simultaneously in use during unixuser garbage collection.

v2021.05-54 (19 January 2025)

The UNIX cache manager changes continue AuriStor's efforts to address a report that was received at the end of September. The end user site uses an OpenAFS cell to distribute software for execution by IBM Spectrum LSF (load sharing facility) managed compute nodes. Each of the approximately one dozen compute nodes is configured with approximately 100 processor cores and 1TB of ram. Each job scheduled by LSF runs in a newly allocated Process Authentication Group (PAG). Some jobs run for hours or days; others complete in a few seconds. LSF has been observed scheduling more than 3000 jobs in under a minute with peak counts of cache manager user objects at 1300. Everything ran well until it didn't. After some unexplained trigger event, all of the remaining free memory on the system would be consumed in about an hour causing the system to panic.

Over the last five months AuriStor has diligently worked to develop theories, optimize the AuriStorFS UNIX cache manager to address each theory; then move on to subsequent ideas once prior theories were disproved. Over this time we have replaced the memory management lifecycle for access rights entries and rx events. We have made it possible for Linux permission checks to be processed without acquiring the GLOCK in the majority of access patterns thereby increasing parallelism. The lifecycle management of PAG allocation and user entity tracking has been rewritten to avoid the GLOCK. The Linux inode lifecycle management has been rewritten which some sites attribute to elimination of swapping on their systems. All of these changes increased throughput and parallelism within and without the cache manager. However, none of them prevented the memory consumption once the unknown trigger event occurred.

It is with great pleasure that we can report that the trigger event and the source of the memory consumption have been identified. The trigger is a damaged Rx packet header which causes the AuriStor Rx stack to place an Rx connection into an RX_PROTOCOL_ERROR error state when there is an in-flight call to an OpenAFS fileserver. The memory consumption is due to the infinite loop that occurs when an Rx connection is in an error state and the error code is not one of the error codes which forces replacement of the Rx connection. Within each pass of the loop a structure is allocated to track the history of per-server errors so they can be logged when hard mount mode is enabled for the cell. AuriStor Rx permits up to 12 calls on each Rx connection. As a result there can be up to a dozen processes each spinning and allocating memory when the Rx connection is placed into the error state.

Days before Christmas 2023 AuriStor identified a race condition which can result in damaged Rx DATA packets being written to the network. Initial transmission of Rx DATA packets can be sent either from an application thread or an Rx listener. Packet retransmission occurs either in an Rx listener thread or in the Rx event thread. The race permits a second thread to modify the Rx packet headers while the sendmsg() syscall from the original transmission is executing. This race is possible because the original thread might exhaust its timeslice during the sendmsg() call and it might be many milliseconds before it is rescheduled on a busy system. In the meantime, packet retransmissions might be initiated either by an Rx listener thread processing an ACK packet or the Rx event thread processing a timeout event. If the initial packet transmission completes while the Rx headers are being rewritten the resulting packet sent on the wire might be garbage. AuriStor observed that if the modified packets were jumbograms that silent data corruption could occur so we disabled all use of jumbograms in v2021.05-35 We also notified both IBM and OpenAFS of the problem.

v2021.05-53 modified the RPC result analyze logic to force the replacement of Rx connections in an error state except when the error signals a token expiration. When a token is expired there is no benefit to replacing the connection until new tokens have been acquired. This release further ensures that infinite memory allocation cannot occur by tracking only the most recent error per fileserver; and only does so if hard mount mode is enabled for the cell.

Processes that issued syscalls which became stuck in a RPC retry loop could not be interrupted. In addition to the unintentional retry loops there are intentional retry loops such as those that occur because a fileserver repeated fails the call with a “VBUSY” error. “VBUSY” originally meant that the required volume was temporarily in use by the volume management service (“volserver”) and would be back in service soon so please wait. Beginning with IBM AFS 3.4 the error code was used to force a retry for reasons having nothing to do with the requested volume’s availability. IBM AFS 3.4 introduced the concept of multi-homed clients which meant that the fileserver could no longer track the identity of a cache manager by its IPv4 address. Instead, each cache manager randomly generates a UUID which the fileserver must query each time an incoming RPC is received on a previously unrecognized Rx connection. An Rx connection might be unrecognized because it’s new, or because the client’s IP address changed, or because the fileserver garbage collected its state information. Regardless of the reason, if a call is received and there is no Rx connection binding to a cache manager UUID, then the fileserver application worker thread to which the call was assigned must issue an outgoing RPC to the client’s RXAFSCB service. Each client has at most one worker thread. Prior to this release, every incoming RXAFSCB call required acquisition of the GLOCK before it could execute. If the GLOCK is unavailable, the RXAFSCB thread blocks and no other pending RXAFSCB calls can be processed. If a RXAFSCB call blocks for too long, the fileserver timeouts trigger and the fileserver replies to the incoming RXAFS call with “VBUSY”. “VBUSY” can also be sent if the fileserver’s quota on the number of calls from the same endpoint is exceeded.

The v2021.05-54 release removes the GLOCK requirement for many RXAFSCB RPCs including all of the UUID query RPCs and all whole volume callback RPCs. Callback RPCs for individual vnodes must still acquire the GLOCK. However, it is hoped that these changes will significantly reduce the time required to process the queries generated due to unrecognized Rx connections. The AuriStor Rx stack permits a dozen calls to be issued on a single logical connection which the fileserver sees as three unique Rx connections from the same endpoint.

Each Unix UID or PAG ID must use an independent Rx connection to each fileserver. On a system which processes /afs syscalls for 1200 UID or PAG IDs in a minute, there is the potential for 3600 Rx connections per fileserver. When the fileserver is OpenAFS, this quickly triggers per endpoint throttling and generates VBUSY storms which can result in unlucky client processes experiencing starvation. With this release, starving processes can be interrupted and RXAFSCB UUID queries can be completed without blocking.

AuriStor believes this release will provide significant benefits for large scale compute nodes, web servers and time sharing systems.

v2021.05-53 (4 January 2025)

v2021.05-52 switched the memory allocation strategy for rx_event objects from a global free list to reliance upon the operating system's memory allocator to allocate individual objects which are freed upon rx_event completion or cancelation. Reliance upon the system allocator introduces the risk of allocation failures due to out of memory conditions. If allocation of an rx_event object fails, a reference count leak on the associated rx_connection or rx_call object can occur preventing garbage collection of the object.
This release handles rx_event allocation failures:
- delayed ACK or delayed ABORT events are replaced by immediate transmission of the ACK or ABORT
- other cases fail the call or connection with an RX_OUT_OF_MEMORY error which is translated on the wire to RX_WIRE_CALL_DEAD (-1).
After executing a call to a fileserver, if the Rx connection is in an error state for any reason other than expired tokens, then schedule the Rx connection to be replaced the next time a call is attempted. All calls issued on an Rx connection that is already in an error state fail immediately preventing further communication with the associated fileserver from the UID/PAG.
Rx connections in an expired token error state are not replaced until new tokens are set.
Rx connections can be placed into an error state either due to local or remote failures. Such failures are often generated by the in-use security class (RXKAD/RXGK) but might be due to an out-of-memory condition, system clock advancement or reversal, or even application errors.
This release includes internal code changes to the management of servers, connections and access rights which will be used by a future release to separate PAG ID allocation from the system's UID namespace.
If 'afsd' is configured with a keytab, receipt of a PoweredOn event will force an attempt to acquire system tokens.
On supported macOS versions, enforce the use of code signing identities to authenticate mach messages involving the AFSBackgrounder and the AFSPreference panel.

v2021.05-52 (5 December 2024)

The v2021.05-50 replacement of the linked list implementation of the vcache access cache with a btree and last lookup cache introduced a memory corruption bug which is fixed in this release.
Eliminate the GLOCK requirement for the collection of function call and RPC statistics collection.
Replace small space allocation with a GLOCK free allocator.
Reimplement the cache manager tracing logging routines so they no longer require the GLOCK.
The Rx networking stack schedules many different types of time sensitive events associated with individual packet retransmission, delayed ack and abort processing, path mtu detection, keep alives, and other categories. Prior to this release, all memory allocated to Rx events objects would not be freed to the operating system until the userspace process terminated or the kernel module was unloaded.
Beginning with this release, instead of having a single, global, free list for rxevents, the system allocator is trusted to be able to allocate memory quickly. This simplifies the Rx implementation, and should be more efficient, as most system allocators will have per thread or core slabs, removing the contention of the global lock protecting the rxevent free list. In addition, the change protects against a network event allocating memory that cannot be readily freed.
On non-Total Store Ordering (TSO)architectures, relaxed atomics used as reference counters do not ensure completion of all prior stores or recency of later loads. This release replaces all read-modify-write atomics to be acquire-release, which, among other things, ensures that only the intended ordering is possible.

v2021.05-51 (26 November 2024)

Fix a vcache (aka inode) reference leak that can occur during an access rights query for a non-directory object if the neither the calling process nor the cache manager possesses tokens for the cell containing the object. The leak was introduced in v2021.05-45.
When scanning the token list for each user searching for expired tokens the user spinlock must be held to prevent races. The updated user management implementation pulled into v2021.05-45 failed to hold the spinlock.
During cache manager module shutdown reset global structures to the module initialization state. Doing so improves the likelihood that the module can be reinitialized.

v2021.05-50 (23 November 2024)

The linked list implementation of the vcache access rights cache was replaced with a btree and a last lookup cache. The btree design is more efficient for lookups and removals, and scales better as the number of uids and/or process authentication groups (pags) accessing /afs increases.
Prune access rights for expired users. Doing so reduces the cost of subsequent lookups.
Cache the access rights obtained for a uid|pag via the "anyAccess" field. Previously, a uid|pag that obtained its rights from the "anyAccess" field had to perform and fail a rights lookup on each and every vcache access.
When setting or clearing tokens for a single cell, no longer invalidate the access rights cached for the uid|pag for all cells. Instead, only invalidate the access rights cached for the cell whose tokens were set or cleared.

v2021.05-49 (16 November 2024)

The output of "tokens" command failed to report yfs-rxgk tokens was broken starting in v2021.05-46

v2021.05-48 (12 November 2024)

Preallocated buffer overflows in XDR responses (CVE-2024-10397)
The AuriStorFS and AFS3 RPC suites rely upon Sun RPC XDR to marshal binary data structures for network transfer. The AuriStor XDR implementation is derived from Sun Microsystems' Sun RPC code base. The Sun RPC XDR API permits memory for output parameters to (optionally) be preallocated which can result in various classes of memory corruption and/or memory leaks in RPC initiator processes.
The AuriStorFS v2021.05-48 release introduces additional data length validation checks within the AuriStor XDR implementation and prohibits the use of preallocated memory for string output parameters or fields. All cache managers, servers and command line tools are modified by these changes.

v2021.05-46 (28 October 2024)

Cache Manager:
- Prevent a kernel memory leak when server preferences are set via the yfs-client.conf [afsd] configuration or via "fs setserverprefs".
- Directory enumeration of a truncated directory now returns an error instead of assuming the end of the directory has been reached.
- Since AFS 3.0, the Unix cache manager has used the root identity credentials to create anonymous outgoing connections to the location service and each fileserver. However, if uid 0 is assigned a token, then those Rx connections will no longer be anonymous. Beginning with this release anonymous outgoing connections are always created with the NOPAG identity (uid 0xffffffff) instead of the root identity.
- When establishing an outgoing rxgk connection, do not fallback to the systemuser's credentials if the user's credentials resulted in a fatal error. Falling back to the systemuser's credentials can result in inappropriate use of an anonymous connection.
- Improved access rights cache correctness for YFS servers
  In prior releases, the access check logic used the file rights for any files fetched from an AuriStorFS fileserver. For files fetched from an AFS-3 fileserver (and, historically, for all files), it used the directory rights, with the (a)dmin right from the file mixed in. The (a)dmin right on a non-directory indicates that the object is owned by the authenticated user.
  This approach has some issues when combined with the access rights cache, and current fileserver callback behavior. On an AuriStorFS file server, the rights on a non-directory may be determined by the rights granted on its parent directory or, with per-file ACLs, those granted on the object itself. The fileserver will only break a non-directory's callback when a per-file ACL is changed - changing a directory ACL will not break callbacks on files within that directory. This means that changing a directory ACL will not invalidate access rights cache entries on files in tha directory, even if the effective ACL on this files has changed, and the cached rights are no longer correct.
  This release works around this by adding a new function which returns the access rights for a file hosted on an AuriStor fileserver. It uses the parent vnode information to locate the parent directory. If the parent directory isn't in the cache, or it doesn't have a valid callback, or if it has been changed since the file's access rights were cached, it clears the current access rights. Files without a parent directory must have per-file ACLs, and so their cached rights can be safely used.
  Note that files with parent vnodes may still have per-file ACLs, and that the breadcrumbing performed by the client may add parent vnode fields to vnodes which don't have them provided by the fileserver. Such vnodes may have their cached access rights cleared more frequently than necessary.
- Add a new mechanism for caching access rights within the vcache structure. This cache is protected via a vcache specific spinlock, and can be accessed without holding the GLOCK.
  This new cache mechanism returns the memory associated with cached rights back to the kernel's slab free memory pool instead of adding the unused rights structures to a cache manager managed free list. The previous cache implementation never returned allocated memory to the kernel. Instead, invalidated access rights were appended to a free access rights queue for later reuse.
- When a volume is accessed via multiple mountpoints a choice must be made regarding which mountpoint is considered to be the active (or parent) mountpoint. This release alters the behavior such that the active mountpoint is set every time a mountpoint is traversed.
  This behavior is easier to understand and is more likely to provide the expected result for a single process that repeatedly accesses volumes from multiple mountpoints. However, it can result in unexpected results when multiple processes are traversing multiple mountpoints in parallel without any synchronization.

v2021.05-45 (Not released)

v2021.05-44a (18 September 2024)

Authentication:
- AuriStorFS v2021.05-44 included an updated version of the Heimdal Kerberos framework used by AuriStorFS when acquiring yfs-rxgk and rxkad authentication tokens. The updated Heimdal included a bug which disabled the use of DNS SRV records for KDC discovery and DNS TXT records for realm discovery. As a side effect token acquisition might fail with an unable to reach any KDC in realm error. This is fixed in v2021.05-44a.

v2021.05-44 (17 August 2024)

Cache Manager:
- Since v0.192 the cache manager has failed acquire the global lock when upgrading a shared-lock to a write-lock during the execution of a background cache chunk file truncation.
Authentication:
- Neither MIT nor Heimdal gssapi nor their gss mechanisms consistently initialize the output 'minorStatus' parameter. Various functions can return either success or failure majorStatus values with minorStatus unassigned. As a result, stack garbage will be used when generating error messages. From now on libyfs_acquire will always initialize the minorStatus output variable to zero before calling into the gssapi library.
Command Parser:
- No longer accept the token "-" as a switch which eventually fails with a CMD_UNKNOWNSWITCH error. Instead, process the token as a data value.
- Optimize the processing of the loop which processes "source" command input.
- If the source command input file is "-", read from stdin.

v2021.05-41 (26 June 2024)

Rx Networking (libyfs_rx):
- A race during event creation can lead to the freeing of the event while its still in use.
- RFC1122 says that Net and Host unreachable ICMP errors might be transient and should therefore not be treated as fatal. There is no such language for the equivalent ICMPV6 errors. However, in practice ICMP6_DST_UNREACH_NOROUTE, ICMP6_DST_UNREACH_BEYONDSCOPE, and ICMP6_DST_UNREACH_ADDR can be transient.
  Linux has considered these ICMPV6 destination unreachable errors as non-fatal going back at least as far as the initial git repository commit.
  AuriStor Rx has always treated these as fatal errors which results in immediate termination of in-flight calls when received. Even if the network route corrects itself before the call timeout period expires. This release mirrors the Linux behavior and makes these errors non-fatal.
Cache Manager:
- For the first time the cache manager can detect the deletion of a volume and handle the creation of a new volume with the same name but a different volume id.
- If the location service reports the deletion of a volume, invalidate all mount points to that volume.
- RXAFS_GetCapabilities RPC failures should not be treated as a fatal error preventing failover to another replica site.
Authentication ("libyfs_acquire") used by aklog, vos, pts, bos, afsio:
- rxkad_k5 token acquisition krb5 ccache management. This release altered the krb5 credential cache management strategy once again to work around different bugs in MIT krb5 and Heimdal.
- New ACQUIRE_ERR_CRED_EXPIRED error code introduced to represent the case when a request for a service credential returns one that is already expired.
Command parser (libyfs_cmd):
- When parsing configuration files there is a depth limit of ten active inclusions. This limit was improperly enforced as a limit of ten included files instead of a depth of ten included files. As of this release it is now possible to populate an includedir directory with any number of .conf files.

v2021.05-40

Not released.

v2021.05-39 (20 May 2024)

Parallel Random Number Generation:
- AuriStorFS processes rely upon the krb5_generate_random() and RAND_bytes() functions to obtain random bytes for cryptographic operations and random counters. krb5_generate_random() internally acquires a mutex to protect internal state information. This mutex has become a significant barrier to the encryption and checksumming of Rx packets with both yfs-rxgk and rxkad.
  
  This release replaces general use of krb5_generate_random() and RAND_bytes() with a per-thread ChaCha20 CS-PRNG. This avoids the acquisition of a global mutex and permits increased parallelism on multi-core systems.
Rx Networking (libyfs_rx):
- The Rx network stack schedules a garbage collection operation to execute once per minute. This operation enforces call timeouts, destroys idle connections and destroys idle peers. The operation has historically been performed by the Rx event thread which is already responsible for performing actions in response to call RTOs, sending NAT Ping and keep-alive packets, and retrying connection challenge and reachability checks.
  
  The time complexity of the garbage collection operation is determined by the number of calls, connections, and peers. The busier the Rx endpoint the more work must be performed during each garbage collection run and the longer it takes to complete. While garbage collection is active other events cannot be processed which can interfere with the proper flow control of active calls.
  
  As with all Rx events, the garbage collection event is scheduled to execute at an absolute clock time. If the system clock drifts (or is administratively set) backwards garbage collection will not be performed until the clock catches up with the scheduled time.
  
  Another responsibility of the garbage collection procedure is to terminate calls if the system clock drifted backwards by five minutes or longer. However, when the clocked drifts backwards garbage collection is not performed until the clock has advanced beyond the point where calls require termination. As a result, calls are not terminated due to backwards clock drift and they can stall.
  
  This release re-implements the garbage collection procedure using a dedicated thread and relative waits. This change ensures that the garbage collection procedure will not prevent the execution of call related events and permits calls to be terminated when large backward clock drifts are detected.
Disk Cache Management:
- Since IBM AFS 3.5, the cache has been considered "too full" even if there exist cache files have been discarded but not yet truncated. When the cache is "too full" most operations that write to the cache will block until truncation of discarded cache files has been performed which results in unnecessary delays. This release fixes the cache such that that discarded but not yet truncated cache files do not block write operations.
- This release permits the cache truncation daemon thread to exit sooner if the cache manager is shutting down.
Improved failover when the RXGK service (co-located with each vlserver) fails to issue tokens. The failures might be the result of misconfiguration, an inability to read keys or loss of Ubik quorum.

v2021.05-38 (29 February 2024)

As with other AuriStorFS releases since the beginning of 2024, this release includes additional improvements to the Rx RPC implementation which are related to the possibility of silent data corruption when Rx jumbograms are in use. Prior releases disabled the negotiation of Rx jumbograms such that the v2021.05-37 Rx peer will refuse to send Rx jumbograms and will request that the remote peer does not send them. However, a bad actor could choose to send Rx jumbograms even though they were asked not to. v2021.05-38 introduces additional protections to ensure that a corrupt Rx jumbogram is dropped instead of being accepted.

The v2021.05-38 Rx RPC implementation also includes two optimizations. First, when Rx initiators complete a call they will no longer send an extra ACK packet to the Rx acceptor of the completed call. The sending of this unnecessary ACK creates additional work for the server which can result in increased latency for other calls being processed by the server.

Second, all AuriStor Rx services require a reach check for incoming calls from Rx peers to help protect against Distributed Reflection Denial of Service (DRDoS) attacks and execution of RPCs when the response cannot be delivered to the caller. A new reach check is required for each new call that arrives more than 60 seconds after the prior reach check completed. v2021.05-38 Rx considers the successful acknowledgment of a response DATA packet as a reach check validation. With this change reach checks will not be periodically required for a peer that completes at least one call per 60 seconds. A 1 RTT delay is therefore avoided each time a reach check can be avoided. In addition, reach checks require the service to process an additional ACK packet. Eliminating a large number of reach checks can improve overall service performance.

The final Rx RPC change in this release is specific to kernel implementations. Prior releases restricted the frequency of executing time scheduled Rx events to a granularity no smaller than 500ms. As a result an RTO timer event for a lost packet could not be shorter than 500ms even if the measured RTT for the connection is significantly smaller. The minimum RTO for a connection in AuriStor Rx is 200ms. The inability to schedule shorter timeouts impacts recovery from packet loss.

v2021.05-37 (5 February 2024)

Rx improvements:
- The v2021.05-36 release permanently disabled all use of Rx jumbograms due to a risk of silent data corruption. However, when advertising the number of acceptable datagrams in the ACK trailer a missing htonl() set the value to 16777216 instead of 1 on little-endian systems.
- When sending a PING ACK as a reachability test, ensure that the previousPacket field is properly assigned to the largest accepted DATA packet sequence number instead of zero.
- Replace the initialization state flag with two flags. One that indicates that Rx initialization began and the other that it succeeded. The first prevents multiple attempts at initialization after failure. The second prevents shutdown from accessing uninitialized structures if initialization failed.
Cache Manager Improvements:
- No longer refuse to start if both the 'cachedir' and 'memcache' options are present in the configuration file.
- New variable to store the maximum number of cache blocks used. which is accessible via /proc/fs/auristorfs/cache/blocks_used_max.

v2021.05-36 (10 January 2024)

Rx improvements:
- Permanently disable all use of Rx jumbograms due to a risk of silent data corruption.
  
  Ever since OpenAFS 1.0, and possibly before, a race condition has existed when Rx transmits packets. As the rx_call.lock is dropped when starting packet transmission, there is no protection for data that is being copied into the kernel by sendmsg(). It is critical that this packet data is not modified by another thread. However, races exist between the application, listener, and event threads that can lead to retransmissions starting whilst an original transmission is still in progress. This can lead to the packet headers being overwritten, and either the original transmission, the retransmission or both sending corrupt data to the peer.
  
  This corruption can affect the packet serial number or packet flags. It is particularly harmful when the packet flags are corrupted, as this can lead to multiple Rx packets which were intended to be sent as Rx jumbograms being delivered and misinterpreted as a single large packet. The eventual result of this depends on the Rx security class in play, but it can cause decrypt integrity errors (rxgk:crypt and rxgk:auth) or corruption of the data stream (rxnull, rxgk:clear or rxkad:auth).
  
  All AuriStorFS servers, OpenAFS 1.6 or later servers, and the Windows cache manager have been shipped with Rx jumbograms disabled by default. The UNIX cache managers however are shipped with jumbograms enabled. There are many AFS cells around the world that continue to deploy OpenAFS 1.4 or earlier fileservers which continue to negotiate the use of Rx jumbograms.
  
  It is worth noting that all AuriStorFS v0.198 and later fileservers and cache managers implement explicit checks that will recognize the corrupted application data stream and prevent corrupted file content from being stored either into an AFS vnode's backing store in the volume's object store or the cache manager's AFS cache. OpenAFS cache managers and fileservers do not have these protections.
  
  With Rx jumbograms disabled the maximum number of Rx packets in a datagram is reduced from 6 to 1; the maximum number of send and receive datagram fragments is reduced from 4 to 1; and the maximum advertised MTU is restricted to 1444 - the maximum rx packet size prior to the introduction of jumbograms in IBM AFS 3.5.
- If the rx call flow state transitions from either the RECOVERY or RESCUE states to the LOSS state as a result of an RTO resend event while writing packets to the network, cease transmission of any new DATA packets if there are packets in the resend queue.
- When the call flow state is LOSS and all packets in the resend queue have been retransmitted and yet the recovery point has not been reached, then permit new DATA packets to be sent in order to maintain a full congestion window.
- Add a safety check to prevent the estimated RTT from underflowing when the actual roundtrip time is smaller than 125us.
- Fix the computation of the padding required for rxgk encrypted packets. This bug results in packets sending 8 bytes fewer per packets than the network permits. This bug accidentally prevented the construction of Rx jumbograms when a call is protected by rxgk:crypt.
- Replace the random number generator with a more security source of random bytes.

v2021.05-33 (27 November 2023)

Rx improvements:
- Not all calls transfer enough data to be able to measure a smoothed round-trip time (SRTT). Calls which are unable to compute a SRTT should not be used to update the peer host RTO value which is used to initialize the RTO for subsequent calls.
  
  Without this change, a single DATA packet call will cause the peer host RTO to be reduced to 0ms. Subsequent calls will start with a RTO value of MAX(0, rxi_minPeerTimeout) where rxi_minPeerTimeout defaults to 200ms. If the actual measured RTO is greater than 200ms, then initial RTO will be too small resulting in premature triggering of the RTO timer and the call flow state entering the loss phase which can significantly hurt performance.
- Initialize the peer host RTO to rxi_minPeerTimeout (which defaults to 200ms) instead of one second. Although RFC6298 recommends the use of one second when no SRTT is available, Rx has long used the rxi_minPeerTimeout value for other purposes which are supposed to be consistent with initial RTO value. It should be noted that Linux TCP uses 200ms instead of one second for this purpose.
- If associating a security class with an Rx connection fails immediately place the Rx connection into an error state. A failure might occur if the security class is unable to access valid key material.
- If an incoming Rx call requires authentication and the security class is unable to successfully generate a challenge, put the incoming Rx connection into an error state and issue an abort to the caller.
- If an incoming Rx call requires authentication and the security class is able to generate a challenge but the challenge cannot be returned to Rx, then treat this as a transient error. Do not acknowledge the incoming DATA packet and do not place the Rx connection into an error state. An attempt to re-issue the challenge will be performed when the DATA packet is retransmitted.
- If an Rx call is terminated due to the expiration of the configured connection dead time, idle dead time, hard dead time, or as a result of clock drift, then send an ABORT to the peer notifying them that the call has been terminated. This is particularly important for terminated outgoing calls. If the peer does not know to terminate the call, then the call channel might be in use when the next outgoing call is issued using the same call channel. If the next incoming call is received by an in-use call channel, the receiver must drop the received DATA packet and return a BUSY packet. The call initiator will need to wait for a retransmission timeout to pass before retransmitting the DATA packet. Receipt of BUSY packets cannot be used to keep a call alive and therefore the requested call is at greater risk of timing out if the network path is congested.
aklog and krb5.log (via libyfs_acquire):
- If the linked Kerberos library implements krb5_cc_cache_match() and libacquire has been told to use an explicit principal name and credential cache, the Kerberos library might return KRB5_CC_NOTFOUND even though the requested credential cache is the correct one to use. This release will not call krb5_cc_cache_match() if the requested credential cache contains the requested principal.
Cell Service Database (cellservdb.conf):
- cellservdb.conf has been synchronized with the 31 Oct 2023 update to the grand.central.org CellServDB file.

v2021.05-32 (9 October 2023)

No significant changes for macOS compared to v2021.05-31

v2021.05-31 (25 September 2023)

New platform:
- macOS 14 Sonoma
macOS 14 Sonoma:

AuriStorFS v2021.05-29 and later installers for macOS 13 Ventura are compatible with macOS 14 Sonoma and do not need to be removed before upgrading to macOS 14 Sonoma. Installation of the macOS 14 Sonoma version of AuriStorFS is recommended.

Cache Manager:
- If an AuriStorFS cache manager is unable to use the yfs-rxgk security class when communicating with an AuriStorFS fileserver, it must assume it is IBM AFS 3.6 or OpenAFS and upgrade it to AuriStorFS if an upgrade probe returns a positive result. Once a fileserver's type is identified as AuriStorFS the type should never be reset; even if communication with the fileserver is lost or the fileserver restarts.
  
  If an AuriStorFS fileserver is replaced by an OpenAFS fileserver on the same endpoint, then the UUID of the OpenAFS must be different. As a result, the OpenAFS fileserver will be observed as distinct from the AuriStorFS fileserver that previously shared the endpoint.
  
  Prior to this release there were circumstances in which the cache manager discarded the fileserver type information and would fail to recognize the fileserver as an AuriStorFS fileserver when yfs-rxgk could not be used. This release prevents the cache manager from resetting the type information if the fileserver is marked down.
- If a fileserver's location service entry is updated with a new uniquifier value (aka version number), this indicates that one of the following might have changed:
  1. the fileserver's capabilities
  2. the fileserver's security policy
  3. the fileserver's knowledge of the cell-wide yfs-rxgk key
  4. the fileserver's endpoints
  Beginning with this release the cache manager will force the establishment of new Rx connections to the fileserver when the uniquifier changes. This ensures that the cache manager will attempt to fetch new per-fileserver yfs-rxgk tokens from the cell's RXGK service, enforce the latest security policy, and not end up in a situation where its existing tokens cannot be used to communicate with the fileserver.
aklog:
- Fix incorrect output when populating the server list for a service fails. The stashed extended error explaining the cause of the failure was not displayed.
- If a cell has neither _afs3-prserver._udp. DNS SRV records nor AFSDB records, the lookup of the cell's protection servers would fail if there is no local cell configuration details. The fallback to use _afs3-vlserver._udp. DNS SRV records did not work. This is corrected in this release.

v2021.05-30 (6 September 2023)

Do not mark a fileserver down in response to a KRB5 error code.
fs cleanacl must not store back to the file server a cleaned acl if it was inherited from a directory. Doing so will create a file acl.
Correct the generation of never expire rxkad_krb5 tokens from Kerberos v5 tickets which must have a start time of Unix epoch and an end time of 0xFFFFFFFF seconds. The incorrectly generated tokens were subject to the maximum lifetime of 30 days.
Correct the generation of the yfs-rxgk RESPONSE packet header which failed to specify the key version generation number used to encrypt the authenticator. If the actual key version is greater than zero, then the authenticator would fail to verify.
Enforce a maximum NAT ping period of 20s to ensure that NAT/PAT/firewall rules due not expire while Rx RPCs are in-flight.

v2021.05-29 (26 June 2023)

Execution of fs commands such as examine, whereis, listquota, fetchacl, cleanacl, storeacl, whoami, lsmount, bypassthreshold and getserverprefs could result in memory leaks by the AuriStorFS kernel extension.

v2021.05-27 (1 May 2023)

Fixes for bugs in vos introduced in v2021.05-26.

v2021.05-26 (17 April 2023)

Fixed a potential kernel memory leak when triggered by fs examine, fs listquota, or fs quota.
Increased logging of VBUSY, VOFFLINE, VSALVAGE, and RX_RESTARTING error responses. A log message is now generated whenever a task begins to wait as a result of one of these error responses from a fileserver. Previously, a message was only logged if the volume location information was expired or discarded.
Several changes to optimize internal volume lookups.
Faster failover to replica sites when a fileserver returns RX_RESTARTING, VNOVOL or VMOVED.
rxdebug regains the ability to report rx call flags and rx_connection flags.
The RXRPC library now terminates calls in the QUEUED state when an ABORT packet is received. This clears the call channel making it available to accept another call and reduces the work load on the worker thread pool.
Fileserver endpoint registration changes no longer result in local invalidation of callbacks from that server.
Receipt of an RXAFSCB_InitCallBackState3 RPC from a fileserver no longer resets the volume site status information for all volumes on all servers.

v2021.05-25 (28 December 2022)

The v2021.05-25 release includes further changes to RXRPC to improve reliability. The changes in this release prevent improper packet size growth. Packet size growth should never occur when a call is attempting to recover from packet loss; and is unsafe when the network path's maximum transmission unit is unknown. Packet size growth with be re-enabled in a future AuriStorFS release that includes Path MTU detection and the Extended SACK functionality.
Improved error text describing the source of invalid values in /etc/yfs/yfs-client.conf or included files and directories.

v2021.05-24 (25 October 2022)

New Platform: macOS 13 (Ventura)

v2021.05-22 (12 September 2022) and v2021.05-21 (6 September 2022)

RX RPC
- If receipt of a DATA packet causes an RX call to enter an error state, do not send the ACK of the DATA packet following the ABORT packet. Only send the ABORT packet.
- AuriStor RX has failed to count and report the number of RX BUSY packets that have been sent. Beginning with this change the sent RX BUSY packet count is once again included in the statistics retrieved via rxdebug server port -rxstats.
- Introduce minimum and maximum bounds checks on the ACK packet trailer fields. If the advertised values are out of bounds for the receiving RX stack, do not abort the call but adjust the values to be consistent with the local RX RPC implementation limits. These changes are necessary to handle broken RX RPC implementations or prevent manipulation by attackers.

New to v2021.05-20 (15 August 2022) and v2021.05-19 (13 August 2022)

RX RPC
- Include the DATA packet serial number in the transmitted reachability check PING ACK. This permits the reachability test ACK to be used for RTT measurement.
- Do not terminate a call due to an idle dead timeout if there is data pending in the receive queue when the timeout period expires. Instead deliver the received data to the application. This change prevents idle dead timeouts on slow lossy network paths.
- Fix assignment of RX DATA, CHALLENGE, and RESPONSE packet serial numbers in macOS (KERNEL). Due to a mistake in the implementation of atomic_add_and_read the wrong serial numbers were assigned to outgoing packets.

New to v2021.05-18 (12 July 2022)

Cache Manager
- Prevent a kernel memory leak of less than 64 bytes for each bulkstat RPC issued to a fileserver. Bulkstat RPCs can be frequently issued and over time this small leak can consume a large amount of kernel memory. Leak introduced in AuriStorFS v0.196.
- The Perl::AFS module directly executes pioctls via the OpenAFS compatibility pioctl interface instead of the AuriStorFS pioctl interface. When Perl::AFS is used to store an access control list (ACL), the deprecated RXAFS_StoreACL RPC would be used in place of the newer RXAFS_StoreACL2 or RXYFS_StoreOpaqueACL2 RPCs. This release alters the behavior of the cache manager to use the newer RPCs if available on the fileserver and fallback to the deprecated RPC. The use of the deprecated RPC was restricted to use of the OpenAFS pioctl interface.
RX RPC
- Handle a race during RX connection pool probes that could have resulted in the wrong RX Service ID being returned for a contacted service. Failure to identify that correct service id can result in a degradation of service.
- The Path MTU detection logic sends padded PING ACK packets and requests a PING_RESPONSE ACK be sent if received. This permits the sender of the PING to probe the maximum transmission unit of the path. Under some circumstances attempts were made to send negative padding which resulted in a failure when sending the PING ACK. As a result, the Path MTU could not be measured. This release prevents the use of negative padding.
Preparation for supporting macOS 13 Ventura when it is released in Fall 2022.
Some shells append a slash to an expanded directory name in response to tab completion. These trailing slashes interfered with "fs lsmount", "fs flushmount" and "fs removeacl" processing. This release includes a change to prevent these commands from breaking when presented a trailing slash.

New to v2021.05-17 (16 May 2022)

Cell Service Database Updates
- Update cern.ch, ics.muni.cz, ifh.de, cs.cmu.edu, qatar.cmu.edu, it.kth.se
- Remove uni-hohenheim.de, rz-uni-jena.de, mathematik.uni-stuttgart.de, stud.mathematik.uni-stuttgart.de, wam.umd.edu
- Add ee.cooper.edu
- Restore ams.cern.ch, md.kth.se, italia
Fix parsing of [afsd] rxwindow configuration which can be used to specified a non-default send/receive RX window size. The current default is 128 packets.
RX Updates
- Add nPacketsReflected

Logging improvements

Cache directory validation errors log messages now include the cache directory path.

Log the active configuration path if "debug" logging is enabled.

More details of rxgk token extraction failures.

New to v2021.05-16 (24 March 2022)

RX - Previous releases re-armed the Retransmission Timeout (RTO) each time a new unacknowledged packet is acknowledged instead of when a new leading edge packet is acknowledged. If leading edge data packet and its retransmission are lost, the call can remain in the "recovery" state where it continues to send new data packets until one of the following is true:
. the maximum window size is reached
. the number of lost and resent packets equals 'cwind'
at which point there is nothing left to transmit. The leading edge data packet can only be retransmitted when entering the "loss" state but since the RTO is reset with each acknowledged packet the call stalls for one RTO period after the last transmitted data packet is acknowledged.

This poor behavior is less noticiable with small window sizes and short lived calls. However, as window sizes and round-trip times increase the impact of a twice lost packet becomes significant.

RX - Never set the high-order bit of the Connection Epoch field. RX peers starting with IBM AFS 3.1b through AuriStor RX v0.191 ignore the source endpoint when matching incoming packets to RX connections if the high-order epoch bit is set. Ignoring the source endpoint is problematic because it can result in a call entering a zombie state whereby all PING ACK packets are immediately responded to the source endpoint of the PING ACK but any delayed ACK or DATA packets are sent to the endpoint bound to the RX connection. An RX client that moves from one network to another or which has a NAT|PAT device between it and the service can find themselves stuck.

Starting with AuriStor RX v0.192 the high-order bit is ignored by AuriStor RX peer when receiving packets. This change to always clear the bit prevents IBM AFS and OpenAFS peers from ignoring the source endpoint.

RX - The initial packetSize calculation for a call is altered to require that all constructed packets before the receipt of the first ACK packet are eligible for use in jumbograms if and only if the local RX stack has jumbograms enabled and the maximum MTU is large enough. By default jumbograms are disabled for all AuriStorFS services. This change will have a beneficial impact if jumbograms are enabled via configuration; or when testing RX performance with "rxperf".

New fs whereis -noresolve option displays the fileservers by network endpoint instead of DNS PTR record hostname.

New to v2021.05-15 (24 January 2022)

kernel - fixed YFS_RXGK service rx connection pool leak

New to v2021.05-14 (20 January 2022)

fs mkmount permit mount point target strings longer than 63 characters.
afsd enhance logging of yfs-rxgk token renewal errors.

afsd gains a "principal = configuration option for use with keytab acquisition of yfs-rxgk tokens for the cache manager identity.

kernel - Avoid unnecessary rx connection replacement by racing threads after token replacement or expiration.

kernel - Fix a regression introduced in v2021.05 where an anonymous combined identity yfs-rxgk token would be replaced after three minutes resulting in the connection switching from yfs-rxgk to rxnull.

kernel - Fix a regression introduced in v0.208 which prevented the invalidation of cached access rights in response to a fileserver callback rpc. The cache would be updated after the first FetchStatus rpc after invalidation.

kernel - Reset combined identity yfs-rxgk tokens when the system token is replaced.

kernel - The replacement of rx connection bundles in the cache manager to permit more than four simultaneous rx calls per uid/pag with trunked rx connections introduced the following regressions in v2021.05.

a memory leak of discarded rx connection objects

failure of NAT ping probes after replacement of an connection

inappropriate use of rx connections after a service upgrade failure

All of these regressions are fixed in patch 14.

New to v2021.05-12 (7 October 2021)

fs ignorelist -type afsmountdir in prior releases could prevent access to /afs.

Location server rpc timeout restored to two minutes instead of twenty minutes.

Location server reachability probe timeout restored to six seconds instead of fifty seconds.

Cell location server upcall results are now cached for fifteen seconds.

Multiple kernel threads waiting for updated cell location server reachability probes now share the results of a single probe.

RX RPC implementation lock hierarchy modified to prevent a lock inversion.

RX RPC client connection reference count leak fixed.

RX RPC deadlock during failed connection service upgrade attempt fixed..

New to v2021.05-9 (25 October 2021)

First public release for macOS 12 Monterey build using XCode 13. When upgrading macOS to Monterey from earlier macOS releases, please upgrade AuriStorFS to v2021.05-9 on the starting macOS release, upgrade to Monterey and then install the Monterey specific v2021.05-9 release.

Improved logging of "afsd" shutdown when "debug" mode is enabled.

Minor RX network stack improvements

New to v2021.05-3 (10 June 2021)

Fix for [cells] cellname = {...} without server list.

New to v2021.05 (31 May 2021)

Multi-homed location servers are finally managed as a single server instead of treating each endpoint as a separate server. The new functionality is a part of the wholesale replacement of the former cell management infrastructure. Location server communication is now entirely managed as a cluster of multi-homed servers for each cell. The new infrastructure does not rely upon the global lock for thread safety.

This release introduces a new infrastructure for managing user/pag entities and tracking their per cell tokens and related connection pools.

Expired tokens are no longer immediately deleted so that its possible for them to be listed by "tokens" for up to two hours.

Prevent a lock inversion introduced in v0.208 that can result in a deadlock involving the GLOCK and the rx call.lock. The deadlock can occur if a cell's list of location servers expires and during the rebuild an rx abort is issued.

Add support for rxkad "auth" mode rx connections in addition to "clear" and "crypt". "auth" mode provides integrity protection without privacy.

Add support for yfs-rxgk "clear" and "auth" rx connection modes.

Do not leak a directory buffer page reference when populating a directory page fails.

Re-initialize state when populating a disk cache entry using the fast path fails and a retry is performed using the slow path. If the data version changes between the attempts it is possible for truncated disk cache data to be treated as valid.

Log warnings if a directory lookup operation fails with an EIO error. An EIO error indicates that an invalid directory header, page header, or directory entry was found.

Do not overwrite RX errors with local errors during Direct-I/O and StoreMini operations. Doing so can result in loss of VBUSY, VOFFLINE, UAENOSPC, and similar errors.

Correct a direct i/o code path which could overwrite a fileserver returned error code with a local RXGEN_CC_UNMARSHAL error.

Correct the StoreMini code path which could overwrite a fileserver returned error code with a local RXGEN_CC_UNMARSHAL error.

Ensure the rx call object is not locked when writing to the network socket.

Removed all knowledge of the KERNEL global lock from RX. Acquiring the GLOCK from RX is never safe if any other lock is held. Doing so is a lock order violation that can result in deadlocks.

Fixed a race in the opr_reservation system that could produce a cache entry reference undercount.

If a directory hash chain contains a circular link, a buffer page reference could be leaked for each traversal.

Each AFS3 directory header and page header contains a magic tag value that can be used in a consistency check but was not previously checked before use of each header. If the header memory is zero filled during a lookup, the search would fail producing an ENOENT error. Starting with this release the magic tag values are validated on each use. An EIO error is returned if there is a tag mismatch.

"fs setcrypt -crypt auth" is now a permitted value. The "auth" mode provides integrity protection but no privacy protection.

Add new "aklog -levels " option which permits requesting "clear" and "auth" modes for use with yfs-rxgk.

Update MKShim to Apple OpenSource MITKerberosShim-79.

Report KLL errors via a notification instead of throwing an exception which (if not caught) will result in process termination.

If an exception occurs while executing "unlog" catch it and ignore it. Otherwise, the process will terminate.

New to v2021.04 (22 April 2021)

Primarily bug fixes for issues that have been present for years.

A possibility of an infinite kernel loop if a rare file write / truncate pattern occurs.

A bug in silly rename handling that can prevent cache manager initiated garbage collection of vnodes.

New to v0.209 (13 March 2021)

fs setserverprefs and fs getserverprefs updated to support IPv6 and CIDR specifications.

Improved error handling during fetch data and store data operations.

Prevents a race between two vfs operations on the same directory which can result in caching of out of date directory contents.

Use cached mount point target information instead of evaluating the mount point's target upon each access.

Avoid rare data cache thrashing condition.

Prevent infinite loop if a disk cache error occurs after the first page in a chunk is written.

Network errors are supposed to be returned to userspace as ETIMEDOUT. Previously some were returned as EIO.

When authentication tokens expire, reissue the fileserver request anonymously. If the anonymous user does not have permission either EACCES or EPERM will be returned as the error to userspace. Previously the vfs request would fail with an RXKADEXPIRED or RXGKEXPIRED error.

If growth of an existing connection vector fails, wait on a call slot in a previously created connection instead of failing the vfs request.

Volume and fileserver location query infrastructure has been replaced with a new modern implementation.

Replace the cache manager's token management infrastructure with a new modern implementation.

New to v0.206 (12 January 2021) - Bug fixes

Prevents a possible panic during unmount of /afs.

Improved failover and retry logic for offline volumes.

New to v0.205 (24 December 2020) - Bug fixes

Volume name-to-id cache improvements

Fix expiration of name-to-id cache entries

Control volume name-to-id via sysctl

Query volume name-to-id statistics via sysctl

Improve error handling for offline volumes

Fix installer to prevent unnecessary installation of Rosetta 2 on Apple Silicon

New to v0.204 (25 November 2020) - Bug fix for macOS Big Sur

v0.204 prevents a kernel panic on Big Sur when AuriStorFS is stopped and restarted without an operating system reboot.

introduces a volume name-to-id cache independent of the volume location cache.

New to v0.203 (13 November 2020) - Bug fix for macOS

v0.203 prevents a potential kernel panic due to network error.

New to v0.201 (12 November 2020) - Universal Big Sur (11.0) release for Apple Silicon and Intel

v0.201 introduces a new cache manager architecture on all macOS versions except for High Sierra (10.12). The new architecture includes a redesign of:

kernel extension load

kernel extension unload (not available on Big Sur)

/afs mount

/afs unmount

userspace networking

The conversion to userspace networking will have two user visible impacts for end users:
The Apple Firewall as configured by System Preferences -> Security & Privacy -> Firewall is now enforced. The "Automatically allow downloaded signed software to receive incoming connections" includes AuriStorFS.

Observed network throughput is likely to vary compared to previous releases.

On Catalina the "Legacy Kernel Extension" warnings that were displayed after boot with previous releases of AuriStorFS are no longer presented with v0.201.

AuriStorFS /afs access is expected to continue to function when upgrading from Mojave or Catalina to Big Sur. However, as AuriStorFS is built specifically for each macOS release, it is recommended that end users install a Big Sur specific AuriStorFS package.
AuriStorFS on Apple Silicon supports hardware accelerated aes256-cts-hmac-sha1-96 and aes128-cts-hmac-sha1-96 using AuriStor's proprietary implementation.

New to v0.200 (4 November 2020) - Final release for macOS El Capitan (10.11)

The network path between a client and a server often traverses one or more network segments separated by NAT/PAT devices. If a NAT/PAT times out a RPCs endpoint translation mid-call, this can result in an extended delay before failure and the server being marked down, or worse, a call that never terminates and a client that appears to hang until the fileserver is restarted.

This release includes significant changes to the RX stack and the UNIX cache manager to detect such conditions, fail the calls quickly and detect when it is safe to retry the RPC.

NAT/PAT devices that drop endpoint mappings while in use are anti-social and can result in unwanted delays and even data loss. Whenever possible they should be avoided when they can be. That said, the changes in this release are a huge step to making the loss of endpoint mappings tolerable.

Fix segmentation fault of Backgrounder when krb5_get_credentials() fails due to lack of network connectivity.

Fix the "afsd" rxbind option which was ignored if the default port, 7001, is in use by another process on the system.

If a direct i/o StoreData or FetchData RPC failed such that it must be retried, the retried RPC would fail due to an attempt to Fetch or Store the wrong amount of data. This is fixed.

Servers are no longer marked down if RPCs fail with RX_CALL_PEER_RESET, RX_CALL_EXCEEDS_WINDOW, or RX_PROTOCOL_ERROR. RPCs that are safe to retry are retried.

Fixed a race between a call entering error state and call completion that can result in a the call remaining in the DALLY state and the connection channel remaining in use. If this occurs during process or system shutdown it can result in a deadlock.

During shutdown cancel any pending delayed aborts to prevent a potential deadlock. If a deadlock occurs when unloading a kernel module a reboot will be required.

Updated cellservdb.conf

New to v0.197.1 (31 August 2020) and v0.198 (10 October 2020)

Prevent Dead vnode has core/unlinkedel/flock panic introduced in v0.197.

New to v0.197 (26 August 2020)

A new callback management framework for UNIX cache managers reduces the expense of processing volume callback RPCs from O(number of vcache objects) to O(1). A significant amount of lock contention has been avoided. The new design reduces the risk of the single callback service worker thread blocking. Delays in processing callbacks on a client can adversely impact fileserver performance and other clients in the cell.

Bulk fetch status RPCs are available on macOS for the first time. Bulk fetch status permits optimistic caching of vnode status information without additional round-trips. Individual fetch status RPCs are no longer issued if a bulk status fails to obtain the required status information.

Hardware accelerated crypto is now available for macOS cache managers. AuriStor's proprietary aes256-cts-hmac-sha1-96 and aes256-cts-hmac-sha512-384 implementations leverage Intel processor extensions: AESNI AVX2 AVX SSE41 SSSE3 to achieve the fastest encrypt, decrypt, sign and verify times for RX packets.

This release optimizes the removal of "._" files that are used to store extended attributes by avoiding unnecessary status fetches when the directory entry is going to be removed.

When removing the final directory entry for an in-use vnode, the directory entry must be silly renamed on the fileserver to prevent removal of the backing vnode. The prior implementation risked blindly renaming over an existing silly rename directory entry.

Behavior change! When the vfs performs a lookup on ".", immediately return the current vnode.

if the object is a mount point, do not perform fakestat and attempt to resolve the target volume root vnode.

do not perform any additional access checks on the vnode. If the caller already knows the vnode the access checks were performed earlier. If the access rights have changed, they will be enforced when the vnode is used just as they would have if the lookup of "." was performed within the vfs.

do not perform a fetch status or fetch data rpcs. Again, the same as if the lookup of "." was performed within the vfs.

Volumes mounted at more than one location in the /afs namespace are problematic on more than one operating system that do not expect directories to have more than one parent. It is particularly problematic if a volume is mounted within itself. Starting with this release any attempt to traverse a mountpoint to the volume containing the mountpoint will fail with ENODEV.

When evaluating volume root vnodes, ensure that the vnode's parent is set to the parent directory of the traversed mountpoint and not the mountpoint. Vnodes without a parent can cause spurious ENOENT errors on Mojave and later.

v0.196 was not publicly released.

New to v0.195 (14 May 2020)

This is a CRITICAL update for AuriStorFS macOS clients.

In Sep 2019 AuriStorFS v0.189 was released which provided faster and less CPU intensive writing of (>64GB) large files to /afs. These improvements introduced a hash collision bug in the store data path of the UNIX cache manager which can result in file corruption. If a hash collision occurs between two or more files that are actively being written to via cached I/O (not direct I/O), dirty data can be discarded from the auristorfs cache before it is written to the fileserver creating a file with a range of zeros (a hole) on the fileserver. This hole might not be visible to the application that wrote the data because the lost data was cached by the operating system. This bug has been fixed in v0.195 and it is for this reason that v0.195 has been designated a CRITICAL release for UNIX/Linux clients.

While debugging a Linux SIGBUS issue, it was observed that receipt of an ICMP network error in response to a transmitted packet could result in termination of an unrelated rx call and could mark a server down. If the terminated call is a StoreData RPC, permanent data loss will occur. All Linux clients derived from the IBM AFS code base experience this bug. The v0.195 release prevents this behavior.

This release includes changes that impact all supported UNIX/Linux cache managers. On macOS there is reduced lock contention between kernel threads when the vcache limit has been reached.

The directory name lookup cache (DNLC) implementation was replaced. The new implementation avoids the use of vcache pointers which did not have associated reference counts, and eliminates the invalidation overhead during callback processing. The DNLC now supports arbitrary directory name lengths; the prior implementation only cached entries with names not exceeding 31 characters.

Prevent matching arbitrary cell name prefixes as aliases. For example "/afs/y" should not be an alias for "your-file-system.com". Some shells, for example "zsh", query the filesystem for names as users type. Delays between typed characters result in filesystem lookups. When this occurs in the /afs dynroot directory, this could result in cellname prefix string matches and the dynamic creation of directory entries for those prefixes.

New to v0.194 (2 April 2020)

This is a CRITICAL release for all macOS users. All prior macOS clients whether AuriStorFS or OpenAFS included a bug that could result in data corruption either when reading or writing.

This release also fixes these other issues:

sign and notarize installer plugin "afscell" bundle. The lack of digital signature prevented the installer from prompting for a cellname on some macOS versions.

prevent potential for corruption when caching locally modified directories.

v0.193 was withdrawn due to a newly introduced bug that could result in data corruption.

New to v0.192 (30 January 2020)

The changes improve stability, efficiency, and scalability. Post-0.189 changes exposed race conditions and reference count errors which can lead to a system panic or deadlock. In addition to addressing these deficiencies this release removes bottlenecks that restricted the number of simultaneous vfs operations that could be processed by the AuriStorFS cache manager. The changes in this release have been successfully tested with greater than 400 simultaneous requests sustained for for several days.

New to v0.191 (16 December 2019)

Restore keyed cache manager capability broken in v0.189.

Add kernel module version string to AuriStorFS Preference Pane.

Other kernel module bug fixes.

New to v0.190 (14 November 2019)

Short-circuit busy volume retries after volume or volume location entry is removed.

New to v0.189 (28 October 2019)

Faster "git status" operation on repositories stored in /afs.

Faster and less CPU intensive writing of (>64GB) large files to /afs. Prior to this release writing files larger than 1TB might not complete. With this release store data throughput is consistent regardless of file size. (See "UNIX Cache Manager large file performance improvements" later in this file).

macOS Catalina (8 October 2019)

AuriStorFS v0.188 released for macOS Catalina (10.15)

New to v0.188 (23 June 2019)

Increased clock resolution for timed waits from 1s to 1ns

Added error handling for rx multi rpcs interrupted by signals

New to v0.186 (29 May 2019)

v0.184 moved the /etc/yfs/cmstate.dat file to /var/yfs. With this change afsd would fail to start if /etc/yfs/cmstate.dat exists but contains invalid state information. This is fixed.

v0.184 introduced a potential deadlock during directory processing. This is fixed.

Handle common error table errors obtained outside an afs_Analyze loop. Map VL errors to ENODEV and RX, RXKAD, RXGK errors to ETIMEDOUT

Log all server down and server up events. Transition events from server probes failed to log messages.

RX RPC networking:

If the RPC initiator successfully completes a call without consuming all of the response data fail the call by sending an RX_PROTOCOL_ERROR ABORT to the acceptor and returning a new error, RX_CALL_PREMATURE_END, to the initiator.
Prior to this change failure to consume all of the response data would be silently ignored by the initiator and the acceptor might resend the unconsumed data until any idle timeout expired. The default idle timeout is 60 seconds.

Avoid transmitting ABORT, CHALLENGE, and RESPONSE packets with an uninitialized sequence number. The sequence number is ignored for these packets but set it to zero.

New to v0.184 (26 March 2019)

The initial congestion window has been reduced from 10 Rx packets to 4. Packet reordering and loss has been observed when sending 10 Rx packets via sendmmsg() in a single burst. The lack of udp packet pacing can also increase the likelihood of transmission stalls due to ack clock variation.

The UNIX Cache Manager underwent major revisions to improve the end user experience by revealing more error codes, improving directory cache efficiency, and overall resiliency. The cache manager implementation was redesigned to be more compatible with operating systems such as Linux and macOS that support restartable system calls. With these changes errors such as "Operation not permitted", "No space left on device", "Quota exceeded", and "Interrupted system call" can be reliably reported to applications. Previously such errors might have been converted to "I/O error".

New to v0.180 (9 November 2018)

RX reliability and performance improvements for high latency and/or lossy network paths such as public wide area networks.

A fix for a macOS firewall triggered kernel panic introduced in v0.177.

New to v0.177 (17 October 2018)

A fix to AuriStor's RX implementation bug introduced in v0.176 that interferes with communication with OpenAFS and IBM Location and File Services.

New to v0.176 (3 October 2018)

AuriStor's RX implementation has undergone a major upgrade of its flow control model. Prior implementations were based on TCP Reno Congestion Control as documented in RFC5681; and SACK behavior that was loosely modelled on RFC2018. The new RX state machine implements SACK based loss recovery as documented in RFC6675, with elements of New Reno from RFC5682 on top of TCP-style congestion control elements as documented in RFC5681. The new RX also implements RFC2861 style congestion window validation.

When sending data the RX peer implementing these changes will be more likely to sustain the maximum available throughput while at the same time improving fairness towards competing network data flows. The improved estimation of available pipe capacity permits an increase in the default maximum window size from 60 packets (84.6 KB) to 128 packets (180.5 KB). The larger window size increases the per call theoretical maximum throughput on a 1ms RTT link from 693 mbit/sec to 1478 mbit/sec and on a 30ms RTT link from 23.1 mbit/sec to 49.39 mbit/sec.

Improve shutdown performance by refusing to give up callbacks to known unreachable file servers and apply a shorter timeout period for the rest.

Permit RXAFSCB_WhoAreYou to be successfully executed after an IBM AFS or OpenAFS fileserver unintentionally requests an RX service upgrade from RXAFSCB to RXYFSCB.

RXAFS timestamps are conveyed in unsigned 32-bit integers with a valid range of 1 Jan 1970 (Unix Epoch) through 07 Feb 2106. UNIX kernel timestamps are stored in 32-bit signed integers with a valid range of 13 Dec 1901 through 19 Jan 2038. This discrepency causes RXAFS timestamps within the 2038-2106 range to display as pre-Epoch.

RX Connection lifecycle management was susceptible to a number of race conditions that could result in assertion failures, the lack of a NAT ping connection to each file server, and the potential reuse of RX connections that should have been discarded.

This release includes a redesigned lifecycle that is thread safe, avoids assertions, prevents NAT ping connection loss, and ensures that discarded connections are not reused.

The 0.174 release unintentionally altered the data structure returned to xstat_cm queries. This release restores the correct wire format.

Since v0.171, if a FetchData RPC fails with a VBUSY error and there is only one reachable fileserver hosting the volume, then the VFS request will immediately with an ETIMEDOUT error ("Connection timed out").

v0.176 corrects three bugs that contributed to this failure condition. One was introduced in v0.171, another in 0.162 and the final one dates to IBM AFS 3.5p1.

The intended behavior is that a cache manager, when all volume sites fail an RPC with a VBUSY error, will sleep for up to 15 seconds and then retry the RPC as if the VBUSY error had never been received. If the RPC continues to receive VBUSY errors from all sites after 100 cycles, the request will be failed with EWOULDBLOCK ("Operation would block") and not ETIMEDOUT.

Prefer VOLMISSING and VOLBUSY error states to network error states when generating error codes to return to the VFS layer. This will result in ENODEV ("No such device") errors when all volume sites return VNOVOL or VOFFLINE errors and EWOULDBLOCK ("Operation would block") errors when all volume sites return VBUSY errors. (v0.176)

New to v0.174 (24 September 2018)

macOS Mojave (10.14) support

New to v0.170 (27 April 2018)

Faster processing of cell configuration information by caching service name to port information.

RX call sequence number rollover to permit calls that require the transmission of more than 5.5TB of data.

Command parser Daylight Saving Time bug fix

Fix a bug that prevented immediate access to a mount point created with "fs mkmount" on the same machine.

Fix the setting of "[afsd] sysnames = " during cache manager startup.

New to v0.168 (6 March 2018)

Corrects "fs setacl -negative" processing [CVE-2018-7168]

Improved reliability for keyed cache managers. More persistent key acquisition renewals.

Major refresh to cellservdb.conf contents.

DNS SRV and DNS AFSDB records now take precedence when use_dns = yes

Kerberos realm hinting provided by
kerberos_realm = [REALM]

DNS host names are resolved instead of reliance on hard coded IP addresses
The cache manager now defaults to sparse dynamic root behavior. Only thiscell and those cells that are assigned aliases are included in /afs directory enumeration at startup. Other cells will be dynamically added upon first access.
Several other quality control improvements.

New to v0.167 (7 December 2017)

Addresses a critical remote denial of service vulnerability [CVE-2017-17432]
Alters the volume location information expiration policy to reduce the risk of single points of failures after volume release operations.
'fs setquota' when issued with quota values larger than 2TB will fail against OpenAFS and IBM AFS file servers
Memory management improvements for the memory caches.

New to v0.164 (11 November 2017)

Internal cache manager redesign. No new functionality.

New to v0.160 (21 September 2017)

Support for OSX High Sierra's new Apple File System (APFS). Customers must upgrade to v0.160 or later before upgrading to OSX High Sierra.
Reduced memory requirements for rx listener thread
Avoid triggering a system panic if an AFS local disk cache file is deleted or becomes inaccessible.
Fixes to "fs" command line output

New to v0.159 (7 August 2017)

Improved failover behavior during volume maintenance operations
Corrected a race that could lead the rx listener thread to enter an infinite loop and cease processing incoming packets.

New to v0.157 (12 July 2017)

Bundled with Heimdal 7.4 to address CVE-2017-11103 (Orpheus' Lyre puts Kerberos to sleep!)
"vos" support for volume quotas larger than 2TB.
"fs flushvolume" works
Fixed a bug that can result in a system panic during server capability testing

New to v0.150

AuriStorFS file server detection improvements

New to v0.149

rxkad encryption is enabled by default. Use "fs setcrypt off" to disable encryption when tokens are available.
Fix a bug in atomic operations on Sierra and El Capitan which could adversely impact Rx behavior.

New to v0.128

Extended attribute ._ files are automatically removed when the associated files are unlinked
Throughput improvements when sending data

New to v0.121

OSX Sierra support

New to v0.117

Cache file moved to a persistent location on local disk
AuriStor File System graphics
Improvements in Background token fetch functionality
Fixed a bug introduced in v0.44 that could result in an operating system crash when enumerating AFS directories containing Unicode file names (v0.106)
El Capitan security changes prevented Finder from deleting files and directories. As of v0.106, the AuriStor OSX client implements the required functionality to permit the DesktopHelperService to securely access the AFS cache as the user permitting Finder to delete files and directories.

Features:

Not vulnerable to OPENAFS-SA-2015-007.
Office 2011 can save to /afs.
Office 2016 can now save files to /afs.
OSX Finder and Preview can open executable documents without triggering a "Corrupted File" warning. .AI, .PDF, .TIFF, .JPG, .DOCX, .XLSX, .PPTX, and other structured documents that might contain scripts were impacted.
All file names are now stored to the file server using Unicode UTF-8 Normalization Form C which is compatible with Microsoft Windows.
All file names are converted to Unicode UTF-8 Normalization Form D for processing by OSX applications.

Known issues:

None

macOS Installer (12.0 Monterey)

Release Notes

Known Issues

If the Kerberos default realm is not configured, a delay of 6m 59s can occur before the AuriStorFS Backgrounder will acquire tokens and display its icon in the macOS menu. This is the result of macOS performing a Bonjour (MDNS) query in an attempt to discover the local realm.
"aklog" will fail to acquire tokens if the current working directory is located in an inaccessible AFS volume. When the "cwd" is inaccessible, the Heimdal framework is unable to initialize the Kerberos v5 context and reports an out of memory error.

New v2021.05-64 (22 Aug 2025)

The v2021.05-64 release fixes a race condition which can lead to a kernel panic when executing the "unlog" command.

v2021.05-63 (31 May 2025)

When AuriStorFS cache managers added support for per-file ACLs in v0.1, the masking of access rights based upon the inode's S_IRWXU mode bits were skipped when reading objects from AuriStorFS fileservers. The masking continued to be applied when reading objects from OpenAFS fileservers. This release applies S_IRWXU mode bits masking of granted access rights to both AuriStorFS and OpenAFS fileservers.
The existing fake stat mountpoint evaluation logic assumes that the fileserver returned a AFSFetchStatus.ParentVnode that refers to the parent directory from which the mountpoint was located. However, this is only guaranteed to be true if the mountpoint does not have a per-file ACL (which is permitted) and has never been cross-directory hard linked (which is also permitted).
This release improves the error handling logic to fail the fake stat evaluation if the AFSFetchStatus.ParentVnode does not match expectations.
This release fixes a metadata inconsistency when a mountpoint is created when disconnected mode is enabled. A mountpoint should be reported to the vfs as a directory and not a symlink.
The cache manager must track for each inode whether the object's status infomration was received from an AFS3 fileserver or an AuriStorFS fileserver. This release alters when that information is recorded to ensure that it is recorded each time the inode's status information is updated by a fileserver response.
This knowledge is required to properly determine how to interpret the status information.
Directory entries referring to mountpoints and volume roots must not be stored within the Directory Name Lookup Cache (DNLC). Instead, full lookups must be performed each time to ensure proper mountpoint resolution.
Prior to this release locally created mountpoints were stored into the DNLC by mistake.

v2021.05-62 (24 April 2025)

macOS Sequoia v15.4 vs Cisco AnyConnect VPN

Beginning with macOS 15.4, use of Cisco AnyConnect VPN breaks DNS SRV, AFSDB and TXT record lookups performed using the cross-platform DNS resolver API. As a result, clients which rely upon DNS-based zero-configuration resolution of Kerberos realms and/or AFS cells database services are unable to access those resources. This release switches the macOS builds to use the Apple DNS API.
The lookup failures impacted all command line tools, clients and services.
Note: Communication failures can occur if the VPN is configured by the organization to block the transmission of fragmented UDP packets. Set [afsd] rxmaxmtu = 1328 to ensure all RX packets are small enough to traverse the VPN link.
This release improves consistency of vnode metadata interpretation when the status information is received via an RPC which does not return a callback.

v2021.05-61 (31 March 2025)

Instead of allocating the same sized hash table for the vcache and dcache objects regardless of how many elements are expected to be stored, scale the hash table sizes. The vcache hash table will scale from 4096 buckets to 131072 buckets with a target of 4 entries per bucket. The dcache hash tables will scale from 4096 buckets to 65536 buckets with a target of 2 entries per bucket.
Historically, afs cache managers which create symlinks (including mountpoints) locally allocate a fake infinite lifetime callback to permit applications to use the symlink (or mountpoint) without requiring FetchStatus RPCs to be issued to the fileserver. This approach has drawbacks because it means that fileservers will not issue a callback break when the volume containing the symlink is taken offline, moved, deleted, etc. It also means that a callback will not be sent if another cache manager removes the symlink.
Beginning with this release the AuriStorFS cache manager will no longer allocate a fake infinite lifetime callback and will instead rely upon fileserver allocated callbacks.
"fs makemount" is supposed to create a mountpoint even if the target volume does not exist or is inaccessible. v2021.05 introduced support for creating mountpoints which kafs which broke the ability to create mountpoints to non-existent volumes.

v2021.05-60 (6 March 2025)

The cache manager retries fileserver RPCs up to 100 times when the fileserver replies with VBUSY, VOFFLINE or VSALVAGE. The cache manager fails the application request after the limit is reached. Prior to this release, the failure would result in the cache manager returning an ENODEV error to the application. But ENODEV indicates that the required volume does not exist and the fileserver has not indicated that the volume does not exist. The cache manager will now return EWOULDBLOCK to the application because the volume might be temporarily inaccessible.
Many OpenAFS and AuriStorFS fileservers can return VNOVOL as a side effect of a race between a volume going offline so that control can be given to the Volume Management Service (volserver) and an incoming RPC requesting that the volume be attached. A VNOVOL can also be returned because the volume was temporarily deleted from the vice partition prior to being replaced by a new clone, or prior to the fileserver marking the volume group as having been moved to another fileserver.
The fileserver replying to an RPC with VNOVOL cannot be trusted to mean that the volume has been permanently removed unless it is matched by the Location Service reporting the Volume Location (VL) entry has been deleted. From now on when a VNOVOL response is received and the VL entry exists, the cache manager will return EWOULDBLOCK instead of ENODEV to the application.
Prior to this release, any VL error code returned from the Location Service would be interpreted as meaning that the requested VL entry does not exist. Starting with this release only the VL_NOENT error will be interpreted as the VL entry does not exist.
Restrict the internal use of the ENOENT error code so the only use of ENOENT is to indicate that a directory entry does not exist. Previously, ENOENT could be returned if a disk cache entry could not be allocated or found. A disk cache related ENOENT returned when processing a directory might have been confused for the requested directory entry not existing.
Restrict the internal use of the ENODEV error code so that the only meaning of ENODEV is that the required volume does not exist. Previously, ENODEV could be returned due to transient errors when failing to contact the Location Service.

v2021.05-59 (13 February 2025)

The fix for garbage collection of idle unixuser objects introduced in v2021.05-56 was incomplete. As a result, Rx client connections might not be garbage collected.

v2021.05-57 (9 February 2025)

No new changes for macOS.

v2021.05-56 (7 February 2025)

Minor bug fixes to the Rx RPC implementation
The fix for garbage collection of idle unixuser objects introduced in v2021.05-55 was incomplete. As a result, Rx client connections might not be garbage collected.

v2021.05-55 (23 January 2025)

Minor bug fixes to the Rx RPC implementation
During garbage collection of idle unixuser objects, any Rx client connections associated with the unixuser (uid or pag) that were in use by another thread would not have their cache manager reference dropped. As a result the Rx client connections could never be garbage collected which is problematic on systems which churn through many unixusers. The behavior has been changed so that unixusers objects with in use Rx connections will not be freed. This permits future attempts to garbage collect unixusers to do so only after the Rx connections can also be freed.
Although this misbehavior has been present for decades, it is only since v2021.05-52 that the problem has been observed. It is possible that the changes to reduce the holding of the GLOCK has created a window of opportunity for connections to be simultaneously in use during unixuser garbage collection.

v2021.05-54 (19 January 2025)

The UNIX cache manager changes continue AuriStor's efforts to address a report that was received at the end of September. The end user site uses an OpenAFS cell to distribute software for execution by IBM Spectrum LSF (load sharing facility) managed compute nodes. Each of the approximately one dozen compute nodes is configured with approximately 100 processor cores and 1TB of ram. Each job scheduled by LSF runs in a newly allocated Process Authentication Group (PAG). Some jobs run for hours or days; others complete in a few seconds. LSF has been observed scheduling more than 3000 jobs in under a minute with peak counts of cache manager user objects at 1300. Everything ran well until it didn't. After some unexplained trigger event, all of the remaining free memory on the system would be consumed in about an hour causing the system to panic.

Over the last five months AuriStor has diligently worked to develop theories, optimize the AuriStorFS UNIX cache manager to address each theory; then move on to subsequent ideas once prior theories were disproved. Over this time we have replaced the memory management lifecycle for access rights entries and rx events. We have made it possible for Linux permission checks to be processed without acquiring the GLOCK in the majority of access patterns thereby increasing parallelism. The lifecycle management of PAG allocation and user entity tracking has been rewritten to avoid the GLOCK. The Linux inode lifecycle management has been rewritten which some sites attribute to elimination of swapping on their systems. All of these changes increased throughput and parallelism within and without the cache manager. However, none of them prevented the memory consumption once the unknown trigger event occurred.

It is with great pleasure that we can report that the trigger event and the source of the memory consumption have been identified. The trigger is a damaged Rx packet header which causes the AuriStor Rx stack to place an Rx connection into an RX_PROTOCOL_ERROR error state when there is an in-flight call to an OpenAFS fileserver. The memory consumption is due to the infinite loop that occurs when an Rx connection is in an error state and the error code is not one of the error codes which forces replacement of the Rx connection. Within each pass of the loop a structure is allocated to track the history of per-server errors so they can be logged when hard mount mode is enabled for the cell. AuriStor Rx permits up to 12 calls on each Rx connection. As a result there can be up to a dozen processes each spinning and allocating memory when the Rx connection is placed into the error state.

Days before Christmas 2023 AuriStor identified a race condition which can result in damaged Rx DATA packets being written to the network. Initial transmission of Rx DATA packets can be sent either from an application thread or an Rx listener. Packet retransmission occurs either in an Rx listener thread or in the Rx event thread. The race permits a second thread to modify the Rx packet headers while the sendmsg() syscall from the original transmission is executing. This race is possible because the original thread might exhaust its timeslice during the sendmsg() call and it might be many milliseconds before it is rescheduled on a busy system. In the meantime, packet retransmissions might be initiated either by an Rx listener thread processing an ACK packet or the Rx event thread processing a timeout event. If the initial packet transmission completes while the Rx headers are being rewritten the resulting packet sent on the wire might be garbage. AuriStor observed that if the modified packets were jumbograms that silent data corruption could occur so we disabled all use of jumbograms in v2021.05-35 We also notified both IBM and OpenAFS of the problem.

v2021.05-53 modified the RPC result analyze logic to force the replacement of Rx connections in an error state except when the error signals a token expiration. When a token is expired there is no benefit to replacing the connection until new tokens have been acquired. This release further ensures that infinite memory allocation cannot occur by tracking only the most recent error per fileserver; and only does so if hard mount mode is enabled for the cell.

Processes that issued syscalls which became stuck in a RPC retry loop could not be interrupted. In addition to the unintentional retry loops there are intentional retry loops such as those that occur because a fileserver repeated fails the call with a “VBUSY” error. “VBUSY” originally meant that the required volume was temporarily in use by the volume management service (“volserver”) and would be back in service soon so please wait. Beginning with IBM AFS 3.4 the error code was used to force a retry for reasons having nothing to do with the requested volume’s availability. IBM AFS 3.4 introduced the concept of multi-homed clients which meant that the fileserver could no longer track the identity of a cache manager by its IPv4 address. Instead, each cache manager randomly generates a UUID which the fileserver must query each time an incoming RPC is received on a previously unrecognized Rx connection. An Rx connection might be unrecognized because it’s new, or because the client’s IP address changed, or because the fileserver garbage collected its state information. Regardless of the reason, if a call is received and there is no Rx connection binding to a cache manager UUID, then the fileserver application worker thread to which the call was assigned must issue an outgoing RPC to the client’s RXAFSCB service. Each client has at most one worker thread. Prior to this release, every incoming RXAFSCB call required acquisition of the GLOCK before it could execute. If the GLOCK is unavailable, the RXAFSCB thread blocks and no other pending RXAFSCB calls can be processed. If a RXAFSCB call blocks for too long, the fileserver timeouts trigger and the fileserver replies to the incoming RXAFS call with “VBUSY”. “VBUSY” can also be sent if the fileserver’s quota on the number of calls from the same endpoint is exceeded.

The v2021.05-54 release removes the GLOCK requirement for many RXAFSCB RPCs including all of the UUID query RPCs and all whole volume callback RPCs. Callback RPCs for individual vnodes must still acquire the GLOCK. However, it is hoped that these changes will significantly reduce the time required to process the queries generated due to unrecognized Rx connections. The AuriStor Rx stack permits a dozen calls to be issued on a single logical connection which the fileserver sees as three unique Rx connections from the same endpoint.

Each Unix UID or PAG ID must use an independent Rx connection to each fileserver. On a system which processes /afs syscalls for 1200 UID or PAG IDs in a minute, there is the potential for 3600 Rx connections per fileserver. When the fileserver is OpenAFS, this quickly triggers per endpoint throttling and generates VBUSY storms which can result in unlucky client processes experiencing starvation. With this release, starving processes can be interrupted and RXAFSCB UUID queries can be completed without blocking.

AuriStor believes this release will provide significant benefits for large scale compute nodes, web servers and time sharing systems.

v2021.05-53 (4 January 2025)

v2021.05-52 switched the memory allocation strategy for rx_event objects from a global free list to reliance upon the operating system's memory allocator to allocate individual objects which are freed upon rx_event completion or cancelation. Reliance upon the system allocator introduces the risk of allocation failures due to out of memory conditions. If allocation of an rx_event object fails, a reference count leak on the associated rx_connection or rx_call object can occur preventing garbage collection of the object.
This release handles rx_event allocation failures:
- delayed ACK or delayed ABORT events are replaced by immediate transmission of the ACK or ABORT
- other cases fail the call or connection with an RX_OUT_OF_MEMORY error which is translated on the wire to RX_WIRE_CALL_DEAD (-1).
After executing a call to a fileserver, if the Rx connection is in an error state for any reason other than expired tokens, then schedule the Rx connection to be replaced the next time a call is attempted. All calls issued on an Rx connection that is already in an error state fail immediately preventing further communication with the associated fileserver from the UID/PAG.
Rx connections in an expired token error state are not replaced until new tokens are set.
Rx connections can be placed into an error state either due to local or remote failures. Such failures are often generated by the in-use security class (RXKAD/RXGK) but might be due to an out-of-memory condition, system clock advancement or reversal, or even application errors.
This release includes internal code changes to the management of servers, connections and access rights which will be used by a future release to separate PAG ID allocation from the system's UID namespace.
If 'afsd' is configured with a keytab, receipt of a PoweredOn event will force an attempt to acquire system tokens.
On supported macOS versions, enforce the use of code signing identities to authenticate mach messages involving the AFSBackgrounder and the AFSPreference panel.

v2021.05-52 (5 December 2024)

The v2021.05-50 replacement of the linked list implementation of the vcache access cache with a btree and last lookup cache introduced a memory corruption bug which is fixed in this release.
Eliminate the GLOCK requirement for the collection of function call and RPC statistics collection.
Replace small space allocation with a GLOCK free allocator.
Reimplement the cache manager tracing logging routines so they no longer require the GLOCK.
The Rx networking stack schedules many different types of time sensitive events associated with individual packet retransmission, delayed ack and abort processing, path mtu detection, keep alives, and other categories. Prior to this release, all memory allocated to Rx events objects would not be freed to the operating system until the userspace process terminated or the kernel module was unloaded.
Beginning with this release, instead of having a single, global, free list for rxevents, the system allocator is trusted to be able to allocate memory quickly. This simplifies the Rx implementation, and should be more efficient, as most system allocators will have per thread or core slabs, removing the contention of the global lock protecting the rxevent free list. In addition, the change protects against a network event allocating memory that cannot be readily freed.
On non-Total Store Ordering (TSO)architectures, relaxed atomics used as reference counters do not ensure completion of all prior stores or recency of later loads. This release replaces all read-modify-write atomics to be acquire-release, which, among other things, ensures that only the intended ordering is possible.

v2021.05-51 (26 November 2024)

Fix a vcache (aka inode) reference leak that can occur during an access rights query for a non-directory object if the neither the calling process nor the cache manager possesses tokens for the cell containing the object. The leak was introduced in v2021.05-45.
When scanning the token list for each user searching for expired tokens the user spinlock must be held to prevent races. The updated user management implementation pulled into v2021.05-45 failed to hold the spinlock.
During cache manager module shutdown reset global structures to the module initialization state. Doing so improves the likelihood that the module can be reinitialized.

v2021.05-50 (23 November 2024)

The linked list implementation of the vcache access rights cache was replaced with a btree and a last lookup cache. The btree design is more efficient for lookups and removals, and scales better as the number of uids and/or process authentication groups (pags) accessing /afs increases.
Prune access rights for expired users. Doing so reduces the cost of subsequent lookups.
Cache the access rights obtained for a uid|pag via the "anyAccess" field. Previously, a uid|pag that obtained its rights from the "anyAccess" field had to perform and fail a rights lookup on each and every vcache access.
When setting or clearing tokens for a single cell, no longer invalidate the access rights cached for the uid|pag for all cells. Instead, only invalidate the access rights cached for the cell whose tokens were set or cleared.

v2021.05-49 (16 November 2024)

The output of "tokens" command failed to report yfs-rxgk tokens was broken starting in v2021.05-46

v2021.05-48 (12 November 2024)

Preallocated buffer overflows in XDR responses (CVE-2024-10397)
The AuriStorFS and AFS3 RPC suites rely upon Sun RPC XDR to marshal binary data structures for network transfer. The AuriStor XDR implementation is derived from Sun Microsystems' Sun RPC code base. The Sun RPC XDR API permits memory for output parameters to (optionally) be preallocated which can result in various classes of memory corruption and/or memory leaks in RPC initiator processes.
The AuriStorFS v2021.05-48 release introduces additional data length validation checks within the AuriStor XDR implementation and prohibits the use of preallocated memory for string output parameters or fields. All cache managers, servers and command line tools are modified by these changes.

v2021.05-46 (28 October 2024)

Cache Manager:
- Prevent a kernel memory leak when server preferences are set via the yfs-client.conf [afsd] configuration or via "fs setserverprefs".
- Directory enumeration of a truncated directory now returns an error instead of assuming the end of the directory has been reached.
- Since AFS 3.0, the Unix cache manager has used the root identity credentials to create anonymous outgoing connections to the location service and each fileserver. However, if uid 0 is assigned a token, then those Rx connections will no longer be anonymous. Beginning with this release anonymous outgoing connections are always created with the NOPAG identity (uid 0xffffffff) instead of the root identity.
- When establishing an outgoing rxgk connection, do not fallback to the systemuser's credentials if the user's credentials resulted in a fatal error. Falling back to the systemuser's credentials can result in inappropriate use of an anonymous connection.
- Improved access rights cache correctness for YFS servers
  In prior releases, the access check logic used the file rights for any files fetched from an AuriStorFS fileserver. For files fetched from an AFS-3 fileserver (and, historically, for all files), it used the directory rights, with the (a)dmin right from the file mixed in. The (a)dmin right on a non-directory indicates that the object is owned by the authenticated user.
  This approach has some issues when combined with the access rights cache, and current fileserver callback behavior. On an AuriStorFS file server, the rights on a non-directory may be determined by the rights granted on its parent directory or, with per-file ACLs, those granted on the object itself. The fileserver will only break a non-directory's callback when a per-file ACL is changed - changing a directory ACL will not break callbacks on files within that directory. This means that changing a directory ACL will not invalidate access rights cache entries on files in tha directory, even if the effective ACL on this files has changed, and the cached rights are no longer correct.
  This release works around this by adding a new function which returns the access rights for a file hosted on an AuriStor fileserver. It uses the parent vnode information to locate the parent directory. If the parent directory isn't in the cache, or it doesn't have a valid callback, or if it has been changed since the file's access rights were cached, it clears the current access rights. Files without a parent directory must have per-file ACLs, and so their cached rights can be safely used.
  Note that files with parent vnodes may still have per-file ACLs, and that the breadcrumbing performed by the client may add parent vnode fields to vnodes which don't have them provided by the fileserver. Such vnodes may have their cached access rights cleared more frequently than necessary.
- Add a new mechanism for caching access rights within the vcache structure. This cache is protected via a vcache specific spinlock, and can be accessed without holding the GLOCK.
  This new cache mechanism returns the memory associated with cached rights back to the kernel's slab free memory pool instead of adding the unused rights structures to a cache manager managed free list. The previous cache implementation never returned allocated memory to the kernel. Instead, invalidated access rights were appended to a free access rights queue for later reuse.
- When a volume is accessed via multiple mountpoints a choice must be made regarding which mountpoint is considered to be the active (or parent) mountpoint. This release alters the behavior such that the active mountpoint is set every time a mountpoint is traversed.
  This behavior is easier to understand and is more likely to provide the expected result for a single process that repeatedly accesses volumes from multiple mountpoints. However, it can result in unexpected results when multiple processes are traversing multiple mountpoints in parallel without any synchronization.

v2021.05-45 (Not released)

v2021.05-44a (18 September 2024)

Authentication:
- AuriStorFS v2021.05-44 included an updated version of the Heimdal Kerberos framework used by AuriStorFS when acquiring yfs-rxgk and rxkad authentication tokens. The updated Heimdal included a bug which disabled the use of DNS SRV records for KDC discovery and DNS TXT records for realm discovery. As a side effect token acquisition might fail with an unable to reach any KDC in realm error. This is fixed in v2021.05-44a.

v2021.05-44 (17 August 2024)

Cache Manager:
- Since v0.192 the cache manager has failed acquire the global lock when upgrading a shared-lock to a write-lock during the execution of a background cache chunk file truncation.
Authentication:
- Neither MIT nor Heimdal gssapi nor their gss mechanisms consistently initialize the output 'minorStatus' parameter. Various functions can return either success or failure majorStatus values with minorStatus unassigned. As a result, stack garbage will be used when generating error messages. From now on libyfs_acquire will always initialize the minorStatus output variable to zero before calling into the gssapi library.
Command Parser:
- No longer accept the token "-" as a switch which eventually fails with a CMD_UNKNOWNSWITCH error. Instead, process the token as a data value.
- Optimize the processing of the loop which processes "source" command input.
- If the source command input file is "-", read from stdin.

v2021.05-41 (26 June 2024)

Rx Networking (libyfs_rx):
- A race during event creation can lead to the freeing of the event while its still in use.
- RFC1122 says that Net and Host unreachable ICMP errors might be transient and should therefore not be treated as fatal. There is no such language for the equivalent ICMPV6 errors. However, in practice ICMP6_DST_UNREACH_NOROUTE, ICMP6_DST_UNREACH_BEYONDSCOPE, and ICMP6_DST_UNREACH_ADDR can be transient.
  Linux has considered these ICMPV6 destination unreachable errors as non-fatal going back at least as far as the initial git repository commit.
  AuriStor Rx has always treated these as fatal errors which results in immediate termination of in-flight calls when received. Even if the network route corrects itself before the call timeout period expires. This release mirrors the Linux behavior and makes these errors non-fatal.
Cache Manager:
- For the first time the cache manager can detect the deletion of a volume and handle the creation of a new volume with the same name but a different volume id.
- If the location service reports the deletion of a volume, invalidate all mount points to that volume.
- RXAFS_GetCapabilities RPC failures should not be treated as a fatal error preventing failover to another replica site.
Authentication ("libyfs_acquire") used by aklog, vos, pts, bos, afsio:
- rxkad_k5 token acquisition krb5 ccache management. This release altered the krb5 credential cache management strategy once again to work around different bugs in MIT krb5 and Heimdal.
- New ACQUIRE_ERR_CRED_EXPIRED error code introduced to represent the case when a request for a service credential returns one that is already expired.
Command parser (libyfs_cmd):
- When parsing configuration files there is a depth limit of ten active inclusions. This limit was improperly enforced as a limit of ten included files instead of a depth of ten included files. As of this release it is now possible to populate an includedir directory with any number of .conf files.

v2021.05-40

Not released.

v2021.05-39 (20 May 2024)

Parallel Random Number Generation:
- AuriStorFS processes rely upon the krb5_generate_random() and RAND_bytes() functions to obtain random bytes for cryptographic operations and random counters. krb5_generate_random() internally acquires a mutex to protect internal state information. This mutex has become a significant barrier to the encryption and checksumming of Rx packets with both yfs-rxgk and rxkad.
  
  This release replaces general use of krb5_generate_random() and RAND_bytes() with a per-thread ChaCha20 CS-PRNG. This avoids the acquisition of a global mutex and permits increased parallelism on multi-core systems.
Rx Networking (libyfs_rx):
- The Rx network stack schedules a garbage collection operation to execute once per minute. This operation enforces call timeouts, destroys idle connections and destroys idle peers. The operation has historically been performed by the Rx event thread which is already responsible for performing actions in response to call RTOs, sending NAT Ping and keep-alive packets, and retrying connection challenge and reachability checks.
  
  The time complexity of the garbage collection operation is determined by the number of calls, connections, and peers. The busier the Rx endpoint the more work must be performed during each garbage collection run and the longer it takes to complete. While garbage collection is active other events cannot be processed which can interfere with the proper flow control of active calls.
  
  As with all Rx events, the garbage collection event is scheduled to execute at an absolute clock time. If the system clock drifts (or is administratively set) backwards garbage collection will not be performed until the clock catches up with the scheduled time.
  
  Another responsibility of the garbage collection procedure is to terminate calls if the system clock drifted backwards by five minutes or longer. However, when the clocked drifts backwards garbage collection is not performed until the clock has advanced beyond the point where calls require termination. As a result, calls are not terminated due to backwards clock drift and they can stall.
  
  This release re-implements the garbage collection procedure using a dedicated thread and relative waits. This change ensures that the garbage collection procedure will not prevent the execution of call related events and permits calls to be terminated when large backward clock drifts are detected.
Disk Cache Management:
- Since IBM AFS 3.5, the cache has been considered "too full" even if there exist cache files have been discarded but not yet truncated. When the cache is "too full" most operations that write to the cache will block until truncation of discarded cache files has been performed which results in unnecessary delays. This release fixes the cache such that that discarded but not yet truncated cache files do not block write operations.
- This release permits the cache truncation daemon thread to exit sooner if the cache manager is shutting down.
Improved failover when the RXGK service (co-located with each vlserver) fails to issue tokens. The failures might be the result of misconfiguration, an inability to read keys or loss of Ubik quorum.

v2021.05-38 (29 February 2024)

As with other AuriStorFS releases since the beginning of 2024, this release includes additional improvements to the Rx RPC implementation which are related to the possibility of silent data corruption when Rx jumbograms are in use. Prior releases disabled the negotiation of Rx jumbograms such that the v2021.05-37 Rx peer will refuse to send Rx jumbograms and will request that the remote peer does not send them. However, a bad actor could choose to send Rx jumbograms even though they were asked not to. v2021.05-38 introduces additional protections to ensure that a corrupt Rx jumbogram is dropped instead of being accepted.

The v2021.05-38 Rx RPC implementation also includes two optimizations. First, when Rx initiators complete a call they will no longer send an extra ACK packet to the Rx acceptor of the completed call. The sending of this unnecessary ACK creates additional work for the server which can result in increased latency for other calls being processed by the server.

Second, all AuriStor Rx services require a reach check for incoming calls from Rx peers to help protect against Distributed Reflection Denial of Service (DRDoS) attacks and execution of RPCs when the response cannot be delivered to the caller. A new reach check is required for each new call that arrives more than 60 seconds after the prior reach check completed. v2021.05-38 Rx considers the successful acknowledgment of a response DATA packet as a reach check validation. With this change reach checks will not be periodically required for a peer that completes at least one call per 60 seconds. A 1 RTT delay is therefore avoided each time a reach check can be avoided. In addition, reach checks require the service to process an additional ACK packet. Eliminating a large number of reach checks can improve overall service performance.

The final Rx RPC change in this release is specific to kernel implementations. Prior releases restricted the frequency of executing time scheduled Rx events to a granularity no smaller than 500ms. As a result an RTO timer event for a lost packet could not be shorter than 500ms even if the measured RTT for the connection is significantly smaller. The minimum RTO for a connection in AuriStor Rx is 200ms. The inability to schedule shorter timeouts impacts recovery from packet loss.

v2021.05-37 (5 February 2024)

Rx improvements:
- The v2021.05-36 release permanently disabled all use of Rx jumbograms due to a risk of silent data corruption. However, when advertising the number of acceptable datagrams in the ACK trailer a missing htonl() set the value to 16777216 instead of 1 on little-endian systems.
- When sending a PING ACK as a reachability test, ensure that the previousPacket field is properly assigned to the largest accepted DATA packet sequence number instead of zero.
- Replace the initialization state flag with two flags. One that indicates that Rx initialization began and the other that it succeeded. The first prevents multiple attempts at initialization after failure. The second prevents shutdown from accessing uninitialized structures if initialization failed.
Cache Manager Improvements:
- No longer refuse to start if both the 'cachedir' and 'memcache' options are present in the configuration file.
- New variable to store the maximum number of cache blocks used. which is accessible via /proc/fs/auristorfs/cache/blocks_used_max.

v2021.05-36 (10 January 2024)

Rx improvements:
- Permanently disable all use of Rx jumbograms due to a risk of silent data corruption.
  
  Ever since OpenAFS 1.0, and possibly before, a race condition has existed when Rx transmits packets. As the rx_call.lock is dropped when starting packet transmission, there is no protection for data that is being copied into the kernel by sendmsg(). It is critical that this packet data is not modified by another thread. However, races exist between the application, listener, and event threads that can lead to retransmissions starting whilst an original transmission is still in progress. This can lead to the packet headers being overwritten, and either the original transmission, the retransmission or both sending corrupt data to the peer.
  
  This corruption can affect the packet serial number or packet flags. It is particularly harmful when the packet flags are corrupted, as this can lead to multiple Rx packets which were intended to be sent as Rx jumbograms being delivered and misinterpreted as a single large packet. The eventual result of this depends on the Rx security class in play, but it can cause decrypt integrity errors (rxgk:crypt and rxgk:auth) or corruption of the data stream (rxnull, rxgk:clear or rxkad:auth).
  
  All AuriStorFS servers, OpenAFS 1.6 or later servers, and the Windows cache manager have been shipped with Rx jumbograms disabled by default. The UNIX cache managers however are shipped with jumbograms enabled. There are many AFS cells around the world that continue to deploy OpenAFS 1.4 or earlier fileservers which continue to negotiate the use of Rx jumbograms.
  
  It is worth noting that all AuriStorFS v0.198 and later fileservers and cache managers implement explicit checks that will recognize the corrupted application data stream and prevent corrupted file content from being stored either into an AFS vnode's backing store in the volume's object store or the cache manager's AFS cache. OpenAFS cache managers and fileservers do not have these protections.
  
  With Rx jumbograms disabled the maximum number of Rx packets in a datagram is reduced from 6 to 1; the maximum number of send and receive datagram fragments is reduced from 4 to 1; and the maximum advertised MTU is restricted to 1444 - the maximum rx packet size prior to the introduction of jumbograms in IBM AFS 3.5.
- If the rx call flow state transitions from either the RECOVERY or RESCUE states to the LOSS state as a result of an RTO resend event while writing packets to the network, cease transmission of any new DATA packets if there are packets in the resend queue.
- When the call flow state is LOSS and all packets in the resend queue have been retransmitted and yet the recovery point has not been reached, then permit new DATA packets to be sent in order to maintain a full congestion window.
- Add a safety check to prevent the estimated RTT from underflowing when the actual roundtrip time is smaller than 125us.
- Fix the computation of the padding required for rxgk encrypted packets. This bug results in packets sending 8 bytes fewer per packets than the network permits. This bug accidentally prevented the construction of Rx jumbograms when a call is protected by rxgk:crypt.
- Replace the random number generator with a more security source of random bytes.

v2021.05-33 (27 November 2023)

Rx improvements:
- Not all calls transfer enough data to be able to measure a smoothed round-trip time (SRTT). Calls which are unable to compute a SRTT should not be used to update the peer host RTO value which is used to initialize the RTO for subsequent calls.
  
  Without this change, a single DATA packet call will cause the peer host RTO to be reduced to 0ms. Subsequent calls will start with a RTO value of MAX(0, rxi_minPeerTimeout) where rxi_minPeerTimeout defaults to 200ms. If the actual measured RTO is greater than 200ms, then initial RTO will be too small resulting in premature triggering of the RTO timer and the call flow state entering the loss phase which can significantly hurt performance.
- Initialize the peer host RTO to rxi_minPeerTimeout (which defaults to 200ms) instead of one second. Although RFC6298 recommends the use of one second when no SRTT is available, Rx has long used the rxi_minPeerTimeout value for other purposes which are supposed to be consistent with initial RTO value. It should be noted that Linux TCP uses 200ms instead of one second for this purpose.
- If associating a security class with an Rx connection fails immediately place the Rx connection into an error state. A failure might occur if the security class is unable to access valid key material.
- If an incoming Rx call requires authentication and the security class is unable to successfully generate a challenge, put the incoming Rx connection into an error state and issue an abort to the caller.
- If an incoming Rx call requires authentication and the security class is able to generate a challenge but the challenge cannot be returned to Rx, then treat this as a transient error. Do not acknowledge the incoming DATA packet and do not place the Rx connection into an error state. An attempt to re-issue the challenge will be performed when the DATA packet is retransmitted.
- If an Rx call is terminated due to the expiration of the configured connection dead time, idle dead time, hard dead time, or as a result of clock drift, then send an ABORT to the peer notifying them that the call has been terminated. This is particularly important for terminated outgoing calls. If the peer does not know to terminate the call, then the call channel might be in use when the next outgoing call is issued using the same call channel. If the next incoming call is received by an in-use call channel, the receiver must drop the received DATA packet and return a BUSY packet. The call initiator will need to wait for a retransmission timeout to pass before retransmitting the DATA packet. Receipt of BUSY packets cannot be used to keep a call alive and therefore the requested call is at greater risk of timing out if the network path is congested.
aklog and krb5.log (via libyfs_acquire):
- If the linked Kerberos library implements krb5_cc_cache_match() and libacquire has been told to use an explicit principal name and credential cache, the Kerberos library might return KRB5_CC_NOTFOUND even though the requested credential cache is the correct one to use. This release will not call krb5_cc_cache_match() if the requested credential cache contains the requested principal.
Cell Service Database (cellservdb.conf):
- cellservdb.conf has been synchronized with the 31 Oct 2023 update to the grand.central.org CellServDB file.

v2021.05-32 (9 October 2023)

No significant changes for macOS compared to v2021.05-31

v2021.05-31 (25 September 2023)

New platform:
- macOS 14 Sonoma
macOS 14 Sonoma:

AuriStorFS v2021.05-29 and later installers for macOS 13 Ventura are compatible with macOS 14 Sonoma and do not need to be removed before upgrading to macOS 14 Sonoma. Installation of the macOS 14 Sonoma version of AuriStorFS is recommended.

Cache Manager:
- If an AuriStorFS cache manager is unable to use the yfs-rxgk security class when communicating with an AuriStorFS fileserver, it must assume it is IBM AFS 3.6 or OpenAFS and upgrade it to AuriStorFS if an upgrade probe returns a positive result. Once a fileserver's type is identified as AuriStorFS the type should never be reset; even if communication with the fileserver is lost or the fileserver restarts.
  
  If an AuriStorFS fileserver is replaced by an OpenAFS fileserver on the same endpoint, then the UUID of the OpenAFS must be different. As a result, the OpenAFS fileserver will be observed as distinct from the AuriStorFS fileserver that previously shared the endpoint.
  
  Prior to this release there were circumstances in which the cache manager discarded the fileserver type information and would fail to recognize the fileserver as an AuriStorFS fileserver when yfs-rxgk could not be used. This release prevents the cache manager from resetting the type information if the fileserver is marked down.
- If a fileserver's location service entry is updated with a new uniquifier value (aka version number), this indicates that one of the following might have changed:
  1. the fileserver's capabilities
  2. the fileserver's security policy
  3. the fileserver's knowledge of the cell-wide yfs-rxgk key
  4. the fileserver's endpoints
  Beginning with this release the cache manager will force the establishment of new Rx connections to the fileserver when the uniquifier changes. This ensures that the cache manager will attempt to fetch new per-fileserver yfs-rxgk tokens from the cell's RXGK service, enforce the latest security policy, and not end up in a situation where its existing tokens cannot be used to communicate with the fileserver.
aklog:
- Fix incorrect output when populating the server list for a service fails. The stashed extended error explaining the cause of the failure was not displayed.
- If a cell has neither _afs3-prserver._udp. DNS SRV records nor AFSDB records, the lookup of the cell's protection servers would fail if there is no local cell configuration details. The fallback to use _afs3-vlserver._udp. DNS SRV records did not work. This is corrected in this release.

v2021.05-30 (6 September 2023)

Do not mark a fileserver down in response to a KRB5 error code.
fs cleanacl must not store back to the file server a cleaned acl if it was inherited from a directory. Doing so will create a file acl.
Correct the generation of never expire rxkad_krb5 tokens from Kerberos v5 tickets which must have a start time of Unix epoch and an end time of 0xFFFFFFFF seconds. The incorrectly generated tokens were subject to the maximum lifetime of 30 days.
Correct the generation of the yfs-rxgk RESPONSE packet header which failed to specify the key version generation number used to encrypt the authenticator. If the actual key version is greater than zero, then the authenticator would fail to verify.
Enforce a maximum NAT ping period of 20s to ensure that NAT/PAT/firewall rules due not expire while Rx RPCs are in-flight.

v2021.05-29 (26 June 2023)

Execution of fs commands such as examine, whereis, listquota, fetchacl, cleanacl, storeacl, whoami, lsmount, bypassthreshold and getserverprefs could result in memory leaks by the AuriStorFS kernel extension.

v2021.05-27 (1 May 2023)

Fixes for bugs in vos introduced in v2021.05-26.

v2021.05-26 (17 April 2023)

Fixed a potential kernel memory leak when triggered by fs examine, fs listquota, or fs quota.
Increased logging of VBUSY, VOFFLINE, VSALVAGE, and RX_RESTARTING error responses. A log message is now generated whenever a task begins to wait as a result of one of these error responses from a fileserver. Previously, a message was only logged if the volume location information was expired or discarded.
Several changes to optimize internal volume lookups.
Faster failover to replica sites when a fileserver returns RX_RESTARTING, VNOVOL or VMOVED.
rxdebug regains the ability to report rx call flags and rx_connection flags.
The RXRPC library now terminates calls in the QUEUED state when an ABORT packet is received. This clears the call channel making it available to accept another call and reduces the work load on the worker thread pool.
Fileserver endpoint registration changes no longer result in local invalidation of callbacks from that server.
Receipt of an RXAFSCB_InitCallBackState3 RPC from a fileserver no longer resets the volume site status information for all volumes on all servers.

v2021.05-25 (28 December 2022)

The v2021.05-25 release includes further changes to RXRPC to improve reliability. The changes in this release prevent improper packet size growth. Packet size growth should never occur when a call is attempting to recover from packet loss; and is unsafe when the network path's maximum transmission unit is unknown. Packet size growth with be re-enabled in a future AuriStorFS release that includes Path MTU detection and the Extended SACK functionality.
Improved error text describing the source of invalid values in /etc/yfs/yfs-client.conf or included files and directories.

v2021.05-24 (25 October 2022)

New Platform: macOS 13 (Ventura)

v2021.05-22 (12 September 2022) and v2021.05-21 (6 September 2022)

RX RPC
- If receipt of a DATA packet causes an RX call to enter an error state, do not send the ACK of the DATA packet following the ABORT packet. Only send the ABORT packet.
- AuriStor RX has failed to count and report the number of RX BUSY packets that have been sent. Beginning with this change the sent RX BUSY packet count is once again included in the statistics retrieved via rxdebug server port -rxstats.
- Introduce minimum and maximum bounds checks on the ACK packet trailer fields. If the advertised values are out of bounds for the receiving RX stack, do not abort the call but adjust the values to be consistent with the local RX RPC implementation limits. These changes are necessary to handle broken RX RPC implementations or prevent manipulation by attackers.

New to v2021.05-20 (15 August 2022) and v2021.05-19 (13 August 2022)

RX RPC
- Include the DATA packet serial number in the transmitted reachability check PING ACK. This permits the reachability test ACK to be used for RTT measurement.
- Do not terminate a call due to an idle dead timeout if there is data pending in the receive queue when the timeout period expires. Instead deliver the received data to the application. This change prevents idle dead timeouts on slow lossy network paths.
- Fix assignment of RX DATA, CHALLENGE, and RESPONSE packet serial numbers in macOS (KERNEL). Due to a mistake in the implementation of atomic_add_and_read the wrong serial numbers were assigned to outgoing packets.

New to v2021.05-18 (12 July 2022)

Cache Manager
- Prevent a kernel memory leak of less than 64 bytes for each bulkstat RPC issued to a fileserver. Bulkstat RPCs can be frequently issued and over time this small leak can consume a large amount of kernel memory. Leak introduced in AuriStorFS v0.196.
- The Perl::AFS module directly executes pioctls via the OpenAFS compatibility pioctl interface instead of the AuriStorFS pioctl interface. When Perl::AFS is used to store an access control list (ACL), the deprecated RXAFS_StoreACL RPC would be used in place of the newer RXAFS_StoreACL2 or RXYFS_StoreOpaqueACL2 RPCs. This release alters the behavior of the cache manager to use the newer RPCs if available on the fileserver and fallback to the deprecated RPC. The use of the deprecated RPC was restricted to use of the OpenAFS pioctl interface.
RX RPC
- Handle a race during RX connection pool probes that could have resulted in the wrong RX Service ID being returned for a contacted service. Failure to identify that correct service id can result in a degradation of service.
- The Path MTU detection logic sends padded PING ACK packets and requests a PING_RESPONSE ACK be sent if received. This permits the sender of the PING to probe the maximum transmission unit of the path. Under some circumstances attempts were made to send negative padding which resulted in a failure when sending the PING ACK. As a result, the Path MTU could not be measured. This release prevents the use of negative padding.
Preparation for supporting macOS 13 Ventura when it is released in Fall 2022.
Some shells append a slash to an expanded directory name in response to tab completion. These trailing slashes interfered with "fs lsmount", "fs flushmount" and "fs removeacl" processing. This release includes a change to prevent these commands from breaking when presented a trailing slash.

New to v2021.05-17 (16 May 2022)

Cell Service Database Updates
- Update cern.ch, ics.muni.cz, ifh.de, cs.cmu.edu, qatar.cmu.edu, it.kth.se
- Remove uni-hohenheim.de, rz-uni-jena.de, mathematik.uni-stuttgart.de, stud.mathematik.uni-stuttgart.de, wam.umd.edu
- Add ee.cooper.edu
- Restore ams.cern.ch, md.kth.se, italia
Fix parsing of [afsd] rxwindow configuration which can be used to specified a non-default send/receive RX window size. The current default is 128 packets.
RX Updates
- Add nPacketsReflected

Logging improvements

Cache directory validation errors log messages now include the cache directory path.

Log the active configuration path if "debug" logging is enabled.

More details of rxgk token extraction failures.

New to v2021.05-16 (24 March 2022)

RX - Previous releases re-armed the Retransmission Timeout (RTO) each time a new unacknowledged packet is acknowledged instead of when a new leading edge packet is acknowledged. If leading edge data packet and its retransmission are lost, the call can remain in the "recovery" state where it continues to send new data packets until one of the following is true:
. the maximum window size is reached
. the number of lost and resent packets equals 'cwind'
at which point there is nothing left to transmit. The leading edge data packet can only be retransmitted when entering the "loss" state but since the RTO is reset with each acknowledged packet the call stalls for one RTO period after the last transmitted data packet is acknowledged.

This poor behavior is less noticiable with small window sizes and short lived calls. However, as window sizes and round-trip times increase the impact of a twice lost packet becomes significant.

RX - Never set the high-order bit of the Connection Epoch field. RX peers starting with IBM AFS 3.1b through AuriStor RX v0.191 ignore the source endpoint when matching incoming packets to RX connections if the high-order epoch bit is set. Ignoring the source endpoint is problematic because it can result in a call entering a zombie state whereby all PING ACK packets are immediately responded to the source endpoint of the PING ACK but any delayed ACK or DATA packets are sent to the endpoint bound to the RX connection. An RX client that moves from one network to another or which has a NAT|PAT device between it and the service can find themselves stuck.

Starting with AuriStor RX v0.192 the high-order bit is ignored by AuriStor RX peer when receiving packets. This change to always clear the bit prevents IBM AFS and OpenAFS peers from ignoring the source endpoint.

RX - The initial packetSize calculation for a call is altered to require that all constructed packets before the receipt of the first ACK packet are eligible for use in jumbograms if and only if the local RX stack has jumbograms enabled and the maximum MTU is large enough. By default jumbograms are disabled for all AuriStorFS services. This change will have a beneficial impact if jumbograms are enabled via configuration; or when testing RX performance with "rxperf".

New fs whereis -noresolve option displays the fileservers by network endpoint instead of DNS PTR record hostname.

New to v2021.05-15 (24 January 2022)

kernel - fixed YFS_RXGK service rx connection pool leak

New to v2021.05-14 (20 January 2022)

fs mkmount permit mount point target strings longer than 63 characters.
afsd enhance logging of yfs-rxgk token renewal errors.

afsd gains a "principal = configuration option for use with keytab acquisition of yfs-rxgk tokens for the cache manager identity.

kernel - Avoid unnecessary rx connection replacement by racing threads after token replacement or expiration.

kernel - Fix a regression introduced in v2021.05 where an anonymous combined identity yfs-rxgk token would be replaced after three minutes resulting in the connection switching from yfs-rxgk to rxnull.

kernel - Fix a regression introduced in v0.208 which prevented the invalidation of cached access rights in response to a fileserver callback rpc. The cache would be updated after the first FetchStatus rpc after invalidation.

kernel - Reset combined identity yfs-rxgk tokens when the system token is replaced.

kernel - The replacement of rx connection bundles in the cache manager to permit more than four simultaneous rx calls per uid/pag with trunked rx connections introduced the following regressions in v2021.05.

a memory leak of discarded rx connection objects

failure of NAT ping probes after replacement of an connection

inappropriate use of rx connections after a service upgrade failure

All of these regressions are fixed in patch 14.

New to v2021.05-12 (7 October 2021)

fs ignorelist -type afsmountdir in prior releases could prevent access to /afs.

Location server rpc timeout restored to two minutes instead of twenty minutes.

Location server reachability probe timeout restored to six seconds instead of fifty seconds.

Cell location server upcall results are now cached for fifteen seconds.

Multiple kernel threads waiting for updated cell location server reachability probes now share the results of a single probe.

RX RPC implementation lock hierarchy modified to prevent a lock inversion.

RX RPC client connection reference count leak fixed.

RX RPC deadlock during failed connection service upgrade attempt fixed..

New to v2021.05-9 (25 October 2021)

First public release for macOS 12 Monterey build using XCode 13. When upgrading macOS to Monterey from earlier macOS releases, please upgrade AuriStorFS to v2021.05-9 on the starting macOS release, upgrade to Monterey and then install the Monterey specific v2021.05-9 release.

Improved logging of "afsd" shutdown when "debug" mode is enabled.

Minor RX network stack improvements

New to v2021.05-3 (10 June 2021)

Fix for [cells] cellname = {...} without server list.

New to v2021.05 (31 May 2021)

Multi-homed location servers are finally managed as a single server instead of treating each endpoint as a separate server. The new functionality is a part of the wholesale replacement of the former cell management infrastructure. Location server communication is now entirely managed as a cluster of multi-homed servers for each cell. The new infrastructure does not rely upon the global lock for thread safety.

This release introduces a new infrastructure for managing user/pag entities and tracking their per cell tokens and related connection pools.

Expired tokens are no longer immediately deleted so that its possible for them to be listed by "tokens" for up to two hours.

Prevent a lock inversion introduced in v0.208 that can result in a deadlock involving the GLOCK and the rx call.lock. The deadlock can occur if a cell's list of location servers expires and during the rebuild an rx abort is issued.

Add support for rxkad "auth" mode rx connections in addition to "clear" and "crypt". "auth" mode provides integrity protection without privacy.

Add support for yfs-rxgk "clear" and "auth" rx connection modes.

Do not leak a directory buffer page reference when populating a directory page fails.

Re-initialize state when populating a disk cache entry using the fast path fails and a retry is performed using the slow path. If the data version changes between the attempts it is possible for truncated disk cache data to be treated as valid.

Log warnings if a directory lookup operation fails with an EIO error. An EIO error indicates that an invalid directory header, page header, or directory entry was found.

Do not overwrite RX errors with local errors during Direct-I/O and StoreMini operations. Doing so can result in loss of VBUSY, VOFFLINE, UAENOSPC, and similar errors.

Correct a direct i/o code path which could overwrite a fileserver returned error code with a local RXGEN_CC_UNMARSHAL error.

Correct the StoreMini code path which could overwrite a fileserver returned error code with a local RXGEN_CC_UNMARSHAL error.

Ensure the rx call object is not locked when writing to the network socket.

Removed all knowledge of the KERNEL global lock from RX. Acquiring the GLOCK from RX is never safe if any other lock is held. Doing so is a lock order violation that can result in deadlocks.

Fixed a race in the opr_reservation system that could produce a cache entry reference undercount.

If a directory hash chain contains a circular link, a buffer page reference could be leaked for each traversal.

Each AFS3 directory header and page header contains a magic tag value that can be used in a consistency check but was not previously checked before use of each header. If the header memory is zero filled during a lookup, the search would fail producing an ENOENT error. Starting with this release the magic tag values are validated on each use. An EIO error is returned if there is a tag mismatch.

"fs setcrypt -crypt auth" is now a permitted value. The "auth" mode provides integrity protection but no privacy protection.

Add new "aklog -levels " option which permits requesting "clear" and "auth" modes for use with yfs-rxgk.

Update MKShim to Apple OpenSource MITKerberosShim-79.

Report KLL errors via a notification instead of throwing an exception which (if not caught) will result in process termination.

If an exception occurs while executing "unlog" catch it and ignore it. Otherwise, the process will terminate.

New to v2021.04 (22 April 2021)

Primarily bug fixes for issues that have been present for years.

A possibility of an infinite kernel loop if a rare file write / truncate pattern occurs.

A bug in silly rename handling that can prevent cache manager initiated garbage collection of vnodes.

New to v0.209 (13 March 2021)

fs setserverprefs and fs getserverprefs updated to support IPv6 and CIDR specifications.

Improved error handling during fetch data and store data operations.

Prevents a race between two vfs operations on the same directory which can result in caching of out of date directory contents.

Use cached mount point target information instead of evaluating the mount point's target upon each access.

Avoid rare data cache thrashing condition.

Prevent infinite loop if a disk cache error occurs after the first page in a chunk is written.

Network errors are supposed to be returned to userspace as ETIMEDOUT. Previously some were returned as EIO.

When authentication tokens expire, reissue the fileserver request anonymously. If the anonymous user does not have permission either EACCES or EPERM will be returned as the error to userspace. Previously the vfs request would fail with an RXKADEXPIRED or RXGKEXPIRED error.

If growth of an existing connection vector fails, wait on a call slot in a previously created connection instead of failing the vfs request.

Volume and fileserver location query infrastructure has been replaced with a new modern implementation.

Replace the cache manager's token management infrastructure with a new modern implementation.

New to v0.206 (12 January 2021) - Bug fixes

Prevents a possible panic during unmount of /afs.

Improved failover and retry logic for offline volumes.

New to v0.205 (24 December 2020) - Bug fixes

Volume name-to-id cache improvements

Fix expiration of name-to-id cache entries

Control volume name-to-id via sysctl

Query volume name-to-id statistics via sysctl

Improve error handling for offline volumes

Fix installer to prevent unnecessary installation of Rosetta 2 on Apple Silicon

New to v0.204 (25 November 2020) - Bug fix for macOS Big Sur

v0.204 prevents a kernel panic on Big Sur when AuriStorFS is stopped and restarted without an operating system reboot.

introduces a volume name-to-id cache independent of the volume location cache.

New to v0.203 (13 November 2020) - Bug fix for macOS

v0.203 prevents a potential kernel panic due to network error.

New to v0.201 (12 November 2020) - Universal Big Sur (11.0) release for Apple Silicon and Intel

v0.201 introduces a new cache manager architecture on all macOS versions except for High Sierra (10.12). The new architecture includes a redesign of:

kernel extension load

kernel extension unload (not available on Big Sur)

/afs mount

/afs unmount

userspace networking

The conversion to userspace networking will have two user visible impacts for end users:
The Apple Firewall as configured by System Preferences -> Security & Privacy -> Firewall is now enforced. The "Automatically allow downloaded signed software to receive incoming connections" includes AuriStorFS.

Observed network throughput is likely to vary compared to previous releases.

On Catalina the "Legacy Kernel Extension" warnings that were displayed after boot with previous releases of AuriStorFS are no longer presented with v0.201.

AuriStorFS /afs access is expected to continue to function when upgrading from Mojave or Catalina to Big Sur. However, as AuriStorFS is built specifically for each macOS release, it is recommended that end users install a Big Sur specific AuriStorFS package.
AuriStorFS on Apple Silicon supports hardware accelerated aes256-cts-hmac-sha1-96 and aes128-cts-hmac-sha1-96 using AuriStor's proprietary implementation.

New to v0.200 (4 November 2020) - Final release for macOS El Capitan (10.11)

The network path between a client and a server often traverses one or more network segments separated by NAT/PAT devices. If a NAT/PAT times out a RPCs endpoint translation mid-call, this can result in an extended delay before failure and the server being marked down, or worse, a call that never terminates and a client that appears to hang until the fileserver is restarted.

This release includes significant changes to the RX stack and the UNIX cache manager to detect such conditions, fail the calls quickly and detect when it is safe to retry the RPC.

NAT/PAT devices that drop endpoint mappings while in use are anti-social and can result in unwanted delays and even data loss. Whenever possible they should be avoided when they can be. That said, the changes in this release are a huge step to making the loss of endpoint mappings tolerable.

Fix segmentation fault of Backgrounder when krb5_get_credentials() fails due to lack of network connectivity.

Fix the "afsd" rxbind option which was ignored if the default port, 7001, is in use by another process on the system.

If a direct i/o StoreData or FetchData RPC failed such that it must be retried, the retried RPC would fail due to an attempt to Fetch or Store the wrong amount of data. This is fixed.

Servers are no longer marked down if RPCs fail with RX_CALL_PEER_RESET, RX_CALL_EXCEEDS_WINDOW, or RX_PROTOCOL_ERROR. RPCs that are safe to retry are retried.

Fixed a race between a call entering error state and call completion that can result in a the call remaining in the DALLY state and the connection channel remaining in use. If this occurs during process or system shutdown it can result in a deadlock.

During shutdown cancel any pending delayed aborts to prevent a potential deadlock. If a deadlock occurs when unloading a kernel module a reboot will be required.

Updated cellservdb.conf

New to v0.197.1 (31 August 2020) and v0.198 (10 October 2020)

Prevent Dead vnode has core/unlinkedel/flock panic introduced in v0.197.

New to v0.197 (26 August 2020)

A new callback management framework for UNIX cache managers reduces the expense of processing volume callback RPCs from O(number of vcache objects) to O(1). A significant amount of lock contention has been avoided. The new design reduces the risk of the single callback service worker thread blocking. Delays in processing callbacks on a client can adversely impact fileserver performance and other clients in the cell.

Bulk fetch status RPCs are available on macOS for the first time. Bulk fetch status permits optimistic caching of vnode status information without additional round-trips. Individual fetch status RPCs are no longer issued if a bulk status fails to obtain the required status information.

Hardware accelerated crypto is now available for macOS cache managers. AuriStor's proprietary aes256-cts-hmac-sha1-96 and aes256-cts-hmac-sha512-384 implementations leverage Intel processor extensions: AESNI AVX2 AVX SSE41 SSSE3 to achieve the fastest encrypt, decrypt, sign and verify times for RX packets.

This release optimizes the removal of "._" files that are used to store extended attributes by avoiding unnecessary status fetches when the directory entry is going to be removed.

When removing the final directory entry for an in-use vnode, the directory entry must be silly renamed on the fileserver to prevent removal of the backing vnode. The prior implementation risked blindly renaming over an existing silly rename directory entry.

Behavior change! When the vfs performs a lookup on ".", immediately return the current vnode.

if the object is a mount point, do not perform fakestat and attempt to resolve the target volume root vnode.

do not perform any additional access checks on the vnode. If the caller already knows the vnode the access checks were performed earlier. If the access rights have changed, they will be enforced when the vnode is used just as they would have if the lookup of "." was performed within the vfs.

do not perform a fetch status or fetch data rpcs. Again, the same as if the lookup of "." was performed within the vfs.

Volumes mounted at more than one location in the /afs namespace are problematic on more than one operating system that do not expect directories to have more than one parent. It is particularly problematic if a volume is mounted within itself. Starting with this release any attempt to traverse a mountpoint to the volume containing the mountpoint will fail with ENODEV.

When evaluating volume root vnodes, ensure that the vnode's parent is set to the parent directory of the traversed mountpoint and not the mountpoint. Vnodes without a parent can cause spurious ENOENT errors on Mojave and later.

v0.196 was not publicly released.

New to v0.195 (14 May 2020)

This is a CRITICAL update for AuriStorFS macOS clients.

In Sep 2019 AuriStorFS v0.189 was released which provided faster and less CPU intensive writing of (>64GB) large files to /afs. These improvements introduced a hash collision bug in the store data path of the UNIX cache manager which can result in file corruption. If a hash collision occurs between two or more files that are actively being written to via cached I/O (not direct I/O), dirty data can be discarded from the auristorfs cache before it is written to the fileserver creating a file with a range of zeros (a hole) on the fileserver. This hole might not be visible to the application that wrote the data because the lost data was cached by the operating system. This bug has been fixed in v0.195 and it is for this reason that v0.195 has been designated a CRITICAL release for UNIX/Linux clients.

While debugging a Linux SIGBUS issue, it was observed that receipt of an ICMP network error in response to a transmitted packet could result in termination of an unrelated rx call and could mark a server down. If the terminated call is a StoreData RPC, permanent data loss will occur. All Linux clients derived from the IBM AFS code base experience this bug. The v0.195 release prevents this behavior.

This release includes changes that impact all supported UNIX/Linux cache managers. On macOS there is reduced lock contention between kernel threads when the vcache limit has been reached.

The directory name lookup cache (DNLC) implementation was replaced. The new implementation avoids the use of vcache pointers which did not have associated reference counts, and eliminates the invalidation overhead during callback processing. The DNLC now supports arbitrary directory name lengths; the prior implementation only cached entries with names not exceeding 31 characters.

Prevent matching arbitrary cell name prefixes as aliases. For example "/afs/y" should not be an alias for "your-file-system.com". Some shells, for example "zsh", query the filesystem for names as users type. Delays between typed characters result in filesystem lookups. When this occurs in the /afs dynroot directory, this could result in cellname prefix string matches and the dynamic creation of directory entries for those prefixes.

New to v0.194 (2 April 2020)

This is a CRITICAL release for all macOS users. All prior macOS clients whether AuriStorFS or OpenAFS included a bug that could result in data corruption either when reading or writing.

This release also fixes these other issues:

sign and notarize installer plugin "afscell" bundle. The lack of digital signature prevented the installer from prompting for a cellname on some macOS versions.

prevent potential for corruption when caching locally modified directories.

v0.193 was withdrawn due to a newly introduced bug that could result in data corruption.

New to v0.192 (30 January 2020)

The changes improve stability, efficiency, and scalability. Post-0.189 changes exposed race conditions and reference count errors which can lead to a system panic or deadlock. In addition to addressing these deficiencies this release removes bottlenecks that restricted the number of simultaneous vfs operations that could be processed by the AuriStorFS cache manager. The changes in this release have been successfully tested with greater than 400 simultaneous requests sustained for for several days.

New to v0.191 (16 December 2019)

Restore keyed cache manager capability broken in v0.189.

Add kernel module version string to AuriStorFS Preference Pane.

Other kernel module bug fixes.

New to v0.190 (14 November 2019)

Short-circuit busy volume retries after volume or volume location entry is removed.

New to v0.189 (28 October 2019)

Faster "git status" operation on repositories stored in /afs.

Faster and less CPU intensive writing of (>64GB) large files to /afs. Prior to this release writing files larger than 1TB might not complete. With this release store data throughput is consistent regardless of file size. (See "UNIX Cache Manager large file performance improvements" later in this file).

macOS Catalina (8 October 2019)

AuriStorFS v0.188 released for macOS Catalina (10.15)

New to v0.188 (23 June 2019)

Increased clock resolution for timed waits from 1s to 1ns

Added error handling for rx multi rpcs interrupted by signals

New to v0.186 (29 May 2019)

v0.184 moved the /etc/yfs/cmstate.dat file to /var/yfs. With this change afsd would fail to start if /etc/yfs/cmstate.dat exists but contains invalid state information. This is fixed.

v0.184 introduced a potential deadlock during directory processing. This is fixed.

Handle common error table errors obtained outside an afs_Analyze loop. Map VL errors to ENODEV and RX, RXKAD, RXGK errors to ETIMEDOUT

Log all server down and server up events. Transition events from server probes failed to log messages.

RX RPC networking:

If the RPC initiator successfully completes a call without consuming all of the response data fail the call by sending an RX_PROTOCOL_ERROR ABORT to the acceptor and returning a new error, RX_CALL_PREMATURE_END, to the initiator.
Prior to this change failure to consume all of the response data would be silently ignored by the initiator and the acceptor might resend the unconsumed data until any idle timeout expired. The default idle timeout is 60 seconds.

Avoid transmitting ABORT, CHALLENGE, and RESPONSE packets with an uninitialized sequence number. The sequence number is ignored for these packets but set it to zero.

New to v0.184 (26 March 2019)

The initial congestion window has been reduced from 10 Rx packets to 4. Packet reordering and loss has been observed when sending 10 Rx packets via sendmmsg() in a single burst. The lack of udp packet pacing can also increase the likelihood of transmission stalls due to ack clock variation.

The UNIX Cache Manager underwent major revisions to improve the end user experience by revealing more error codes, improving directory cache efficiency, and overall resiliency. The cache manager implementation was redesigned to be more compatible with operating systems such as Linux and macOS that support restartable system calls. With these changes errors such as "Operation not permitted", "No space left on device", "Quota exceeded", and "Interrupted system call" can be reliably reported to applications. Previously such errors might have been converted to "I/O error".

New to v0.180 (9 November 2018)

RX reliability and performance improvements for high latency and/or lossy network paths such as public wide area networks.

A fix for a macOS firewall triggered kernel panic introduced in v0.177.

New to v0.177 (17 October 2018)

A fix to AuriStor's RX implementation bug introduced in v0.176 that interferes with communication with OpenAFS and IBM Location and File Services.

New to v0.176 (3 October 2018)

AuriStor's RX implementation has undergone a major upgrade of its flow control model. Prior implementations were based on TCP Reno Congestion Control as documented in RFC5681; and SACK behavior that was loosely modelled on RFC2018. The new RX state machine implements SACK based loss recovery as documented in RFC6675, with elements of New Reno from RFC5682 on top of TCP-style congestion control elements as documented in RFC5681. The new RX also implements RFC2861 style congestion window validation.

When sending data the RX peer implementing these changes will be more likely to sustain the maximum available throughput while at the same time improving fairness towards competing network data flows. The improved estimation of available pipe capacity permits an increase in the default maximum window size from 60 packets (84.6 KB) to 128 packets (180.5 KB). The larger window size increases the per call theoretical maximum throughput on a 1ms RTT link from 693 mbit/sec to 1478 mbit/sec and on a 30ms RTT link from 23.1 mbit/sec to 49.39 mbit/sec.

Improve shutdown performance by refusing to give up callbacks to known unreachable file servers and apply a shorter timeout period for the rest.

Permit RXAFSCB_WhoAreYou to be successfully executed after an IBM AFS or OpenAFS fileserver unintentionally requests an RX service upgrade from RXAFSCB to RXYFSCB.

RXAFS timestamps are conveyed in unsigned 32-bit integers with a valid range of 1 Jan 1970 (Unix Epoch) through 07 Feb 2106. UNIX kernel timestamps are stored in 32-bit signed integers with a valid range of 13 Dec 1901 through 19 Jan 2038. This discrepency causes RXAFS timestamps within the 2038-2106 range to display as pre-Epoch.

RX Connection lifecycle management was susceptible to a number of race conditions that could result in assertion failures, the lack of a NAT ping connection to each file server, and the potential reuse of RX connections that should have been discarded.

This release includes a redesigned lifecycle that is thread safe, avoids assertions, prevents NAT ping connection loss, and ensures that discarded connections are not reused.

The 0.174 release unintentionally altered the data structure returned to xstat_cm queries. This release restores the correct wire format.

Since v0.171, if a FetchData RPC fails with a VBUSY error and there is only one reachable fileserver hosting the volume, then the VFS request will immediately with an ETIMEDOUT error ("Connection timed out").

v0.176 corrects three bugs that contributed to this failure condition. One was introduced in v0.171, another in 0.162 and the final one dates to IBM AFS 3.5p1.

The intended behavior is that a cache manager, when all volume sites fail an RPC with a VBUSY error, will sleep for up to 15 seconds and then retry the RPC as if the VBUSY error had never been received. If the RPC continues to receive VBUSY errors from all sites after 100 cycles, the request will be failed with EWOULDBLOCK ("Operation would block") and not ETIMEDOUT.

Prefer VOLMISSING and VOLBUSY error states to network error states when generating error codes to return to the VFS layer. This will result in ENODEV ("No such device") errors when all volume sites return VNOVOL or VOFFLINE errors and EWOULDBLOCK ("Operation would block") errors when all volume sites return VBUSY errors. (v0.176)

New to v0.174 (24 September 2018)

macOS Mojave (10.14) support

New to v0.170 (27 April 2018)

Faster processing of cell configuration information by caching service name to port information.

RX call sequence number rollover to permit calls that require the transmission of more than 5.5TB of data.

Command parser Daylight Saving Time bug fix

Fix a bug that prevented immediate access to a mount point created with "fs mkmount" on the same machine.

Fix the setting of "[afsd] sysnames = " during cache manager startup.

New to v0.168 (6 March 2018)

Corrects "fs setacl -negative" processing [CVE-2018-7168]

Improved reliability for keyed cache managers. More persistent key acquisition renewals.

Major refresh to cellservdb.conf contents.

DNS SRV and DNS AFSDB records now take precedence when use_dns = yes

Kerberos realm hinting provided by
kerberos_realm = [REALM]

DNS host names are resolved instead of reliance on hard coded IP addresses
The cache manager now defaults to sparse dynamic root behavior. Only thiscell and those cells that are assigned aliases are included in /afs directory enumeration at startup. Other cells will be dynamically added upon first access.
Several other quality control improvements.

New to v0.167 (7 December 2017)

Addresses a critical remote denial of service vulnerability [CVE-2017-17432]
Alters the volume location information expiration policy to reduce the risk of single points of failures after volume release operations.
'fs setquota' when issued with quota values larger than 2TB will fail against OpenAFS and IBM AFS file servers
Memory management improvements for the memory caches.

New to v0.164 (11 November 2017)

Internal cache manager redesign. No new functionality.

New to v0.160 (21 September 2017)

Support for OSX High Sierra's new Apple File System (APFS). Customers must upgrade to v0.160 or later before upgrading to OSX High Sierra.
Reduced memory requirements for rx listener thread
Avoid triggering a system panic if an AFS local disk cache file is deleted or becomes inaccessible.
Fixes to "fs" command line output

New to v0.159 (7 August 2017)

Improved failover behavior during volume maintenance operations
Corrected a race that could lead the rx listener thread to enter an infinite loop and cease processing incoming packets.

New to v0.157 (12 July 2017)

Bundled with Heimdal 7.4 to address CVE-2017-11103 (Orpheus' Lyre puts Kerberos to sleep!)
"vos" support for volume quotas larger than 2TB.
"fs flushvolume" works
Fixed a bug that can result in a system panic during server capability testing

New to v0.150

AuriStorFS file server detection improvements

New to v0.149

rxkad encryption is enabled by default. Use "fs setcrypt off" to disable encryption when tokens are available.
Fix a bug in atomic operations on Sierra and El Capitan which could adversely impact Rx behavior.

New to v0.128

Extended attribute ._ files are automatically removed when the associated files are unlinked
Throughput improvements when sending data

New to v0.121

OSX Sierra support

New to v0.117

Cache file moved to a persistent location on local disk
AuriStor File System graphics
Improvements in Background token fetch functionality
Fixed a bug introduced in v0.44 that could result in an operating system crash when enumerating AFS directories containing Unicode file names (v0.106)
El Capitan security changes prevented Finder from deleting files and directories. As of v0.106, the AuriStor OSX client implements the required functionality to permit the DesktopHelperService to securely access the AFS cache as the user permitting Finder to delete files and directories.

Features:

Not vulnerable to OPENAFS-SA-2015-007.
Office 2011 can save to /afs.
Office 2016 can now save files to /afs.
OSX Finder and Preview can open executable documents without triggering a "Corrupted File" warning. .AI, .PDF, .TIFF, .JPG, .DOCX, .XLSX, .PPTX, and other structured documents that might contain scripts were impacted.
All file names are now stored to the file server using Unicode UTF-8 Normalization Form C which is compatible with Microsoft Windows.
All file names are converted to Unicode UTF-8 Normalization Form D for processing by OSX applications.

Known issues:

None

macOS Installer (11.0 Big Sur)

Release Notes

Known Issues

If the Kerberos default realm is not configured, a delay of 6m 59s can occur before the AuriStorFS Backgrounder will acquire tokens and display its icon in the macOS menu. This is the result of macOS performing a Bonjour (MDNS) query in an attempt to discover the local realm.
"aklog" will fail to acquire tokens if the current working directory is located in an inaccessible AFS volume. When the "cwd" is inaccessible, the Heimdal framework is unable to initialize the Kerberos v5 context and reports an out of memory error.

New v2021.05-64 (22 Aug 2025)

The v2021.05-64 release fixes a race condition which can lead to a kernel panic when executing the "unlog" command.

v2021.05-63 (31 May 2025)

When AuriStorFS cache managers added support for per-file ACLs in v0.1, the masking of access rights based upon the inode's S_IRWXU mode bits were skipped when reading objects from AuriStorFS fileservers. The masking continued to be applied when reading objects from OpenAFS fileservers. This release applies S_IRWXU mode bits masking of granted access rights to both AuriStorFS and OpenAFS fileservers.
The existing fake stat mountpoint evaluation logic assumes that the fileserver returned a AFSFetchStatus.ParentVnode that refers to the parent directory from which the mountpoint was located. However, this is only guaranteed to be true if the mountpoint does not have a per-file ACL (which is permitted) and has never been cross-directory hard linked (which is also permitted).
This release improves the error handling logic to fail the fake stat evaluation if the AFSFetchStatus.ParentVnode does not match expectations.
This release fixes a metadata inconsistency when a mountpoint is created when disconnected mode is enabled. A mountpoint should be reported to the vfs as a directory and not a symlink.
The cache manager must track for each inode whether the object's status infomration was received from an AFS3 fileserver or an AuriStorFS fileserver. This release alters when that information is recorded to ensure that it is recorded each time the inode's status information is updated by a fileserver response.
This knowledge is required to properly determine how to interpret the status information.
Directory entries referring to mountpoints and volume roots must not be stored within the Directory Name Lookup Cache (DNLC). Instead, full lookups must be performed each time to ensure proper mountpoint resolution.
Prior to this release locally created mountpoints were stored into the DNLC by mistake.

v2021.05-62 (24 April 2025)

macOS Sequoia v15.4 vs Cisco AnyConnect VPN

Beginning with macOS 15.4, use of Cisco AnyConnect VPN breaks DNS SRV, AFSDB and TXT record lookups performed using the cross-platform DNS resolver API. As a result, clients which rely upon DNS-based zero-configuration resolution of Kerberos realms and/or AFS cells database services are unable to access those resources. This release switches the macOS builds to use the Apple DNS API.
The lookup failures impacted all command line tools, clients and services.
Note: Communication failures can occur if the VPN is configured by the organization to block the transmission of fragmented UDP packets. Set [afsd] rxmaxmtu = 1328 to ensure all RX packets are small enough to traverse the VPN link.
This release improves consistency of vnode metadata interpretation when the status information is received via an RPC which does not return a callback.

v2021.05-61 (31 March 2025)

Instead of allocating the same sized hash table for the vcache and dcache objects regardless of how many elements are expected to be stored, scale the hash table sizes. The vcache hash table will scale from 4096 buckets to 131072 buckets with a target of 4 entries per bucket. The dcache hash tables will scale from 4096 buckets to 65536 buckets with a target of 2 entries per bucket.
Historically, afs cache managers which create symlinks (including mountpoints) locally allocate a fake infinite lifetime callback to permit applications to use the symlink (or mountpoint) without requiring FetchStatus RPCs to be issued to the fileserver. This approach has drawbacks because it means that fileservers will not issue a callback break when the volume containing the symlink is taken offline, moved, deleted, etc. It also means that a callback will not be sent if another cache manager removes the symlink.
Beginning with this release the AuriStorFS cache manager will no longer allocate a fake infinite lifetime callback and will instead rely upon fileserver allocated callbacks.
"fs makemount" is supposed to create a mountpoint even if the target volume does not exist or is inaccessible. v2021.05 introduced support for creating mountpoints which kafs which broke the ability to create mountpoints to non-existent volumes.

v2021.05-60 (6 March 2025)

The cache manager retries fileserver RPCs up to 100 times when the fileserver replies with VBUSY, VOFFLINE or VSALVAGE. The cache manager fails the application request after the limit is reached. Prior to this release, the failure would result in the cache manager returning an ENODEV error to the application. But ENODEV indicates that the required volume does not exist and the fileserver has not indicated that the volume does not exist. The cache manager will now return EWOULDBLOCK to the application because the volume might be temporarily inaccessible.
Many OpenAFS and AuriStorFS fileservers can return VNOVOL as a side effect of a race between a volume going offline so that control can be given to the Volume Management Service (volserver) and an incoming RPC requesting that the volume be attached. A VNOVOL can also be returned because the volume was temporarily deleted from the vice partition prior to being replaced by a new clone, or prior to the fileserver marking the volume group as having been moved to another fileserver.
The fileserver replying to an RPC with VNOVOL cannot be trusted to mean that the volume has been permanently removed unless it is matched by the Location Service reporting the Volume Location (VL) entry has been deleted. From now on when a VNOVOL response is received and the VL entry exists, the cache manager will return EWOULDBLOCK instead of ENODEV to the application.
Prior to this release, any VL error code returned from the Location Service would be interpreted as meaning that the requested VL entry does not exist. Starting with this release only the VL_NOENT error will be interpreted as the VL entry does not exist.
Restrict the internal use of the ENOENT error code so the only use of ENOENT is to indicate that a directory entry does not exist. Previously, ENOENT could be returned if a disk cache entry could not be allocated or found. A disk cache related ENOENT returned when processing a directory might have been confused for the requested directory entry not existing.
Restrict the internal use of the ENODEV error code so that the only meaning of ENODEV is that the required volume does not exist. Previously, ENODEV could be returned due to transient errors when failing to contact the Location Service.

v2021.05-59 (13 February 2025)

The fix for garbage collection of idle unixuser objects introduced in v2021.05-56 was incomplete. As a result, Rx client connections might not be garbage collected.

v2021.05-57 (9 February 2025)

No new changes for macOS.

v2021.05-56 (7 February 2025)

Minor bug fixes to the Rx RPC implementation
The fix for garbage collection of idle unixuser objects introduced in v2021.05-55 was incomplete. As a result, Rx client connections might not be garbage collected.

v2021.05-55 (23 January 2025)

Minor bug fixes to the Rx RPC implementation
During garbage collection of idle unixuser objects, any Rx client connections associated with the unixuser (uid or pag) that were in use by another thread would not have their cache manager reference dropped. As a result the Rx client connections could never be garbage collected which is problematic on systems which churn through many unixusers. The behavior has been changed so that unixusers objects with in use Rx connections will not be freed. This permits future attempts to garbage collect unixusers to do so only after the Rx connections can also be freed.
Although this misbehavior has been present for decades, it is only since v2021.05-52 that the problem has been observed. It is possible that the changes to reduce the holding of the GLOCK has created a window of opportunity for connections to be simultaneously in use during unixuser garbage collection.

v2021.05-54 (19 January 2025)

The UNIX cache manager changes continue AuriStor's efforts to address a report that was received at the end of September. The end user site uses an OpenAFS cell to distribute software for execution by IBM Spectrum LSF (load sharing facility) managed compute nodes. Each of the approximately one dozen compute nodes is configured with approximately 100 processor cores and 1TB of ram. Each job scheduled by LSF runs in a newly allocated Process Authentication Group (PAG). Some jobs run for hours or days; others complete in a few seconds. LSF has been observed scheduling more than 3000 jobs in under a minute with peak counts of cache manager user objects at 1300. Everything ran well until it didn't. After some unexplained trigger event, all of the remaining free memory on the system would be consumed in about an hour causing the system to panic.

Over the last five months AuriStor has diligently worked to develop theories, optimize the AuriStorFS UNIX cache manager to address each theory; then move on to subsequent ideas once prior theories were disproved. Over this time we have replaced the memory management lifecycle for access rights entries and rx events. We have made it possible for Linux permission checks to be processed without acquiring the GLOCK in the majority of access patterns thereby increasing parallelism. The lifecycle management of PAG allocation and user entity tracking has been rewritten to avoid the GLOCK. The Linux inode lifecycle management has been rewritten which some sites attribute to elimination of swapping on their systems. All of these changes increased throughput and parallelism within and without the cache manager. However, none of them prevented the memory consumption once the unknown trigger event occurred.

It is with great pleasure that we can report that the trigger event and the source of the memory consumption have been identified. The trigger is a damaged Rx packet header which causes the AuriStor Rx stack to place an Rx connection into an RX_PROTOCOL_ERROR error state when there is an in-flight call to an OpenAFS fileserver. The memory consumption is due to the infinite loop that occurs when an Rx connection is in an error state and the error code is not one of the error codes which forces replacement of the Rx connection. Within each pass of the loop a structure is allocated to track the history of per-server errors so they can be logged when hard mount mode is enabled for the cell. AuriStor Rx permits up to 12 calls on each Rx connection. As a result there can be up to a dozen processes each spinning and allocating memory when the Rx connection is placed into the error state.

Days before Christmas 2023 AuriStor identified a race condition which can result in damaged Rx DATA packets being written to the network. Initial transmission of Rx DATA packets can be sent either from an application thread or an Rx listener. Packet retransmission occurs either in an Rx listener thread or in the Rx event thread. The race permits a second thread to modify the Rx packet headers while the sendmsg() syscall from the original transmission is executing. This race is possible because the original thread might exhaust its timeslice during the sendmsg() call and it might be many milliseconds before it is rescheduled on a busy system. In the meantime, packet retransmissions might be initiated either by an Rx listener thread processing an ACK packet or the Rx event thread processing a timeout event. If the initial packet transmission completes while the Rx headers are being rewritten the resulting packet sent on the wire might be garbage. AuriStor observed that if the modified packets were jumbograms that silent data corruption could occur so we disabled all use of jumbograms in v2021.05-35 We also notified both IBM and OpenAFS of the problem.

v2021.05-53 modified the RPC result analyze logic to force the replacement of Rx connections in an error state except when the error signals a token expiration. When a token is expired there is no benefit to replacing the connection until new tokens have been acquired. This release further ensures that infinite memory allocation cannot occur by tracking only the most recent error per fileserver; and only does so if hard mount mode is enabled for the cell.

Processes that issued syscalls which became stuck in a RPC retry loop could not be interrupted. In addition to the unintentional retry loops there are intentional retry loops such as those that occur because a fileserver repeated fails the call with a “VBUSY” error. “VBUSY” originally meant that the required volume was temporarily in use by the volume management service (“volserver”) and would be back in service soon so please wait. Beginning with IBM AFS 3.4 the error code was used to force a retry for reasons having nothing to do with the requested volume’s availability. IBM AFS 3.4 introduced the concept of multi-homed clients which meant that the fileserver could no longer track the identity of a cache manager by its IPv4 address. Instead, each cache manager randomly generates a UUID which the fileserver must query each time an incoming RPC is received on a previously unrecognized Rx connection. An Rx connection might be unrecognized because it’s new, or because the client’s IP address changed, or because the fileserver garbage collected its state information. Regardless of the reason, if a call is received and there is no Rx connection binding to a cache manager UUID, then the fileserver application worker thread to which the call was assigned must issue an outgoing RPC to the client’s RXAFSCB service. Each client has at most one worker thread. Prior to this release, every incoming RXAFSCB call required acquisition of the GLOCK before it could execute. If the GLOCK is unavailable, the RXAFSCB thread blocks and no other pending RXAFSCB calls can be processed. If a RXAFSCB call blocks for too long, the fileserver timeouts trigger and the fileserver replies to the incoming RXAFS call with “VBUSY”. “VBUSY” can also be sent if the fileserver’s quota on the number of calls from the same endpoint is exceeded.

The v2021.05-54 release removes the GLOCK requirement for many RXAFSCB RPCs including all of the UUID query RPCs and all whole volume callback RPCs. Callback RPCs for individual vnodes must still acquire the GLOCK. However, it is hoped that these changes will significantly reduce the time required to process the queries generated due to unrecognized Rx connections. The AuriStor Rx stack permits a dozen calls to be issued on a single logical connection which the fileserver sees as three unique Rx connections from the same endpoint.

Each Unix UID or PAG ID must use an independent Rx connection to each fileserver. On a system which processes /afs syscalls for 1200 UID or PAG IDs in a minute, there is the potential for 3600 Rx connections per fileserver. When the fileserver is OpenAFS, this quickly triggers per endpoint throttling and generates VBUSY storms which can result in unlucky client processes experiencing starvation. With this release, starving processes can be interrupted and RXAFSCB UUID queries can be completed without blocking.

AuriStor believes this release will provide significant benefits for large scale compute nodes, web servers and time sharing systems.

v2021.05-53 (4 January 2025)

v2021.05-52 switched the memory allocation strategy for rx_event objects from a global free list to reliance upon the operating system's memory allocator to allocate individual objects which are freed upon rx_event completion or cancelation. Reliance upon the system allocator introduces the risk of allocation failures due to out of memory conditions. If allocation of an rx_event object fails, a reference count leak on the associated rx_connection or rx_call object can occur preventing garbage collection of the object.
This release handles rx_event allocation failures:
- delayed ACK or delayed ABORT events are replaced by immediate transmission of the ACK or ABORT
- other cases fail the call or connection with an RX_OUT_OF_MEMORY error which is translated on the wire to RX_WIRE_CALL_DEAD (-1).
After executing a call to a fileserver, if the Rx connection is in an error state for any reason other than expired tokens, then schedule the Rx connection to be replaced the next time a call is attempted. All calls issued on an Rx connection that is already in an error state fail immediately preventing further communication with the associated fileserver from the UID/PAG.
Rx connections in an expired token error state are not replaced until new tokens are set.
Rx connections can be placed into an error state either due to local or remote failures. Such failures are often generated by the in-use security class (RXKAD/RXGK) but might be due to an out-of-memory condition, system clock advancement or reversal, or even application errors.
This release includes internal code changes to the management of servers, connections and access rights which will be used by a future release to separate PAG ID allocation from the system's UID namespace.
If 'afsd' is configured with a keytab, receipt of a PoweredOn event will force an attempt to acquire system tokens.
On supported macOS versions, enforce the use of code signing identities to authenticate mach messages involving the AFSBackgrounder and the AFSPreference panel.

v2021.05-52 (5 December 2024)

The v2021.05-50 replacement of the linked list implementation of the vcache access cache with a btree and last lookup cache introduced a memory corruption bug which is fixed in this release.
Eliminate the GLOCK requirement for the collection of function call and RPC statistics collection.
Replace small space allocation with a GLOCK free allocator.
Reimplement the cache manager tracing logging routines so they no longer require the GLOCK.
The Rx networking stack schedules many different types of time sensitive events associated with individual packet retransmission, delayed ack and abort processing, path mtu detection, keep alives, and other categories. Prior to this release, all memory allocated to Rx events objects would not be freed to the operating system until the userspace process terminated or the kernel module was unloaded.
Beginning with this release, instead of having a single, global, free list for rxevents, the system allocator is trusted to be able to allocate memory quickly. This simplifies the Rx implementation, and should be more efficient, as most system allocators will have per thread or core slabs, removing the contention of the global lock protecting the rxevent free list. In addition, the change protects against a network event allocating memory that cannot be readily freed.
On non-Total Store Ordering (TSO)architectures, relaxed atomics used as reference counters do not ensure completion of all prior stores or recency of later loads. This release replaces all read-modify-write atomics to be acquire-release, which, among other things, ensures that only the intended ordering is possible.

v2021.05-51 (26 November 2024)

Fix a vcache (aka inode) reference leak that can occur during an access rights query for a non-directory object if the neither the calling process nor the cache manager possesses tokens for the cell containing the object. The leak was introduced in v2021.05-45.
When scanning the token list for each user searching for expired tokens the user spinlock must be held to prevent races. The updated user management implementation pulled into v2021.05-45 failed to hold the spinlock.
During cache manager module shutdown reset global structures to the module initialization state. Doing so improves the likelihood that the module can be reinitialized.

v2021.05-50 (23 November 2024)

The linked list implementation of the vcache access rights cache was replaced with a btree and a last lookup cache. The btree design is more efficient for lookups and removals, and scales better as the number of uids and/or process authentication groups (pags) accessing /afs increases.
Prune access rights for expired users. Doing so reduces the cost of subsequent lookups.
Cache the access rights obtained for a uid|pag via the "anyAccess" field. Previously, a uid|pag that obtained its rights from the "anyAccess" field had to perform and fail a rights lookup on each and every vcache access.
When setting or clearing tokens for a single cell, no longer invalidate the access rights cached for the uid|pag for all cells. Instead, only invalidate the access rights cached for the cell whose tokens were set or cleared.

v2021.05-49 (16 November 2024)

The output of "tokens" command failed to report yfs-rxgk tokens was broken starting in v2021.05-46

v2021.05-48 (12 November 2024)

Preallocated buffer overflows in XDR responses (CVE-2024-10397)
The AuriStorFS and AFS3 RPC suites rely upon Sun RPC XDR to marshal binary data structures for network transfer. The AuriStor XDR implementation is derived from Sun Microsystems' Sun RPC code base. The Sun RPC XDR API permits memory for output parameters to (optionally) be preallocated which can result in various classes of memory corruption and/or memory leaks in RPC initiator processes.
The AuriStorFS v2021.05-48 release introduces additional data length validation checks within the AuriStor XDR implementation and prohibits the use of preallocated memory for string output parameters or fields. All cache managers, servers and command line tools are modified by these changes.

v2021.05-46 (28 October 2024)

Cache Manager:
- Prevent a kernel memory leak when server preferences are set via the yfs-client.conf [afsd] configuration or via "fs setserverprefs".
- Directory enumeration of a truncated directory now returns an error instead of assuming the end of the directory has been reached.
- Since AFS 3.0, the Unix cache manager has used the root identity credentials to create anonymous outgoing connections to the location service and each fileserver. However, if uid 0 is assigned a token, then those Rx connections will no longer be anonymous. Beginning with this release anonymous outgoing connections are always created with the NOPAG identity (uid 0xffffffff) instead of the root identity.
- When establishing an outgoing rxgk connection, do not fallback to the systemuser's credentials if the user's credentials resulted in a fatal error. Falling back to the systemuser's credentials can result in inappropriate use of an anonymous connection.
- Improved access rights cache correctness for YFS servers
  In prior releases, the access check logic used the file rights for any files fetched from an AuriStorFS fileserver. For files fetched from an AFS-3 fileserver (and, historically, for all files), it used the directory rights, with the (a)dmin right from the file mixed in. The (a)dmin right on a non-directory indicates that the object is owned by the authenticated user.
  This approach has some issues when combined with the access rights cache, and current fileserver callback behavior. On an AuriStorFS file server, the rights on a non-directory may be determined by the rights granted on its parent directory or, with per-file ACLs, those granted on the object itself. The fileserver will only break a non-directory's callback when a per-file ACL is changed - changing a directory ACL will not break callbacks on files within that directory. This means that changing a directory ACL will not invalidate access rights cache entries on files in tha directory, even if the effective ACL on this files has changed, and the cached rights are no longer correct.
  This release works around this by adding a new function which returns the access rights for a file hosted on an AuriStor fileserver. It uses the parent vnode information to locate the parent directory. If the parent directory isn't in the cache, or it doesn't have a valid callback, or if it has been changed since the file's access rights were cached, it clears the current access rights. Files without a parent directory must have per-file ACLs, and so their cached rights can be safely used.
  Note that files with parent vnodes may still have per-file ACLs, and that the breadcrumbing performed by the client may add parent vnode fields to vnodes which don't have them provided by the fileserver. Such vnodes may have their cached access rights cleared more frequently than necessary.
- Add a new mechanism for caching access rights within the vcache structure. This cache is protected via a vcache specific spinlock, and can be accessed without holding the GLOCK.
  This new cache mechanism returns the memory associated with cached rights back to the kernel's slab free memory pool instead of adding the unused rights structures to a cache manager managed free list. The previous cache implementation never returned allocated memory to the kernel. Instead, invalidated access rights were appended to a free access rights queue for later reuse.
- When a volume is accessed via multiple mountpoints a choice must be made regarding which mountpoint is considered to be the active (or parent) mountpoint. This release alters the behavior such that the active mountpoint is set every time a mountpoint is traversed.
  This behavior is easier to understand and is more likely to provide the expected result for a single process that repeatedly accesses volumes from multiple mountpoints. However, it can result in unexpected results when multiple processes are traversing multiple mountpoints in parallel without any synchronization.

v2021.05-45 (Not released)

v2021.05-44a (18 September 2024)

Authentication:
- AuriStorFS v2021.05-44 included an updated version of the Heimdal Kerberos framework used by AuriStorFS when acquiring yfs-rxgk and rxkad authentication tokens. The updated Heimdal included a bug which disabled the use of DNS SRV records for KDC discovery and DNS TXT records for realm discovery. As a side effect token acquisition might fail with an unable to reach any KDC in realm error. This is fixed in v2021.05-44a.

v2021.05-44 (17 August 2024)

Cache Manager:
- Since v0.192 the cache manager has failed acquire the global lock when upgrading a shared-lock to a write-lock during the execution of a background cache chunk file truncation.
Authentication:
- Neither MIT nor Heimdal gssapi nor their gss mechanisms consistently initialize the output 'minorStatus' parameter. Various functions can return either success or failure majorStatus values with minorStatus unassigned. As a result, stack garbage will be used when generating error messages. From now on libyfs_acquire will always initialize the minorStatus output variable to zero before calling into the gssapi library.
Command Parser:
- No longer accept the token "-" as a switch which eventually fails with a CMD_UNKNOWNSWITCH error. Instead, process the token as a data value.
- Optimize the processing of the loop which processes "source" command input.
- If the source command input file is "-", read from stdin.

v2021.05-41 (26 June 2024)

Rx Networking (libyfs_rx):
- A race during event creation can lead to the freeing of the event while its still in use.
- RFC1122 says that Net and Host unreachable ICMP errors might be transient and should therefore not be treated as fatal. There is no such language for the equivalent ICMPV6 errors. However, in practice ICMP6_DST_UNREACH_NOROUTE, ICMP6_DST_UNREACH_BEYONDSCOPE, and ICMP6_DST_UNREACH_ADDR can be transient.
  Linux has considered these ICMPV6 destination unreachable errors as non-fatal going back at least as far as the initial git repository commit.
  AuriStor Rx has always treated these as fatal errors which results in immediate termination of in-flight calls when received. Even if the network route corrects itself before the call timeout period expires. This release mirrors the Linux behavior and makes these errors non-fatal.
Cache Manager:
- For the first time the cache manager can detect the deletion of a volume and handle the creation of a new volume with the same name but a different volume id.
- If the location service reports the deletion of a volume, invalidate all mount points to that volume.
- RXAFS_GetCapabilities RPC failures should not be treated as a fatal error preventing failover to another replica site.
Authentication ("libyfs_acquire") used by aklog, vos, pts, bos, afsio:
- rxkad_k5 token acquisition krb5 ccache management. This release altered the krb5 credential cache management strategy once again to work around different bugs in MIT krb5 and Heimdal.
- New ACQUIRE_ERR_CRED_EXPIRED error code introduced to represent the case when a request for a service credential returns one that is already expired.
Command parser (libyfs_cmd):
- When parsing configuration files there is a depth limit of ten active inclusions. This limit was improperly enforced as a limit of ten included files instead of a depth of ten included files. As of this release it is now possible to populate an includedir directory with any number of .conf files.

v2021.05-40

Not released.

v2021.05-39 (20 May 2024)

Parallel Random Number Generation:
- AuriStorFS processes rely upon the krb5_generate_random() and RAND_bytes() functions to obtain random bytes for cryptographic operations and random counters. krb5_generate_random() internally acquires a mutex to protect internal state information. This mutex has become a significant barrier to the encryption and checksumming of Rx packets with both yfs-rxgk and rxkad.
  
  This release replaces general use of krb5_generate_random() and RAND_bytes() with a per-thread ChaCha20 CS-PRNG. This avoids the acquisition of a global mutex and permits increased parallelism on multi-core systems.
Rx Networking (libyfs_rx):
- The Rx network stack schedules a garbage collection operation to execute once per minute. This operation enforces call timeouts, destroys idle connections and destroys idle peers. The operation has historically been performed by the Rx event thread which is already responsible for performing actions in response to call RTOs, sending NAT Ping and keep-alive packets, and retrying connection challenge and reachability checks.
  
  The time complexity of the garbage collection operation is determined by the number of calls, connections, and peers. The busier the Rx endpoint the more work must be performed during each garbage collection run and the longer it takes to complete. While garbage collection is active other events cannot be processed which can interfere with the proper flow control of active calls.
  
  As with all Rx events, the garbage collection event is scheduled to execute at an absolute clock time. If the system clock drifts (or is administratively set) backwards garbage collection will not be performed until the clock catches up with the scheduled time.
  
  Another responsibility of the garbage collection procedure is to terminate calls if the system clock drifted backwards by five minutes or longer. However, when the clocked drifts backwards garbage collection is not performed until the clock has advanced beyond the point where calls require termination. As a result, calls are not terminated due to backwards clock drift and they can stall.
  
  This release re-implements the garbage collection procedure using a dedicated thread and relative waits. This change ensures that the garbage collection procedure will not prevent the execution of call related events and permits calls to be terminated when large backward clock drifts are detected.
Disk Cache Management:
- Since IBM AFS 3.5, the cache has been considered "too full" even if there exist cache files have been discarded but not yet truncated. When the cache is "too full" most operations that write to the cache will block until truncation of discarded cache files has been performed which results in unnecessary delays. This release fixes the cache such that that discarded but not yet truncated cache files do not block write operations.
- This release permits the cache truncation daemon thread to exit sooner if the cache manager is shutting down.
Improved failover when the RXGK service (co-located with each vlserver) fails to issue tokens. The failures might be the result of misconfiguration, an inability to read keys or loss of Ubik quorum.

v2021.05-38 (29 February 2024)

As with other AuriStorFS releases since the beginning of 2024, this release includes additional improvements to the Rx RPC implementation which are related to the possibility of silent data corruption when Rx jumbograms are in use. Prior releases disabled the negotiation of Rx jumbograms such that the v2021.05-37 Rx peer will refuse to send Rx jumbograms and will request that the remote peer does not send them. However, a bad actor could choose to send Rx jumbograms even though they were asked not to. v2021.05-38 introduces additional protections to ensure that a corrupt Rx jumbogram is dropped instead of being accepted.

The v2021.05-38 Rx RPC implementation also includes two optimizations. First, when Rx initiators complete a call they will no longer send an extra ACK packet to the Rx acceptor of the completed call. The sending of this unnecessary ACK creates additional work for the server which can result in increased latency for other calls being processed by the server.

Second, all AuriStor Rx services require a reach check for incoming calls from Rx peers to help protect against Distributed Reflection Denial of Service (DRDoS) attacks and execution of RPCs when the response cannot be delivered to the caller. A new reach check is required for each new call that arrives more than 60 seconds after the prior reach check completed. v2021.05-38 Rx considers the successful acknowledgment of a response DATA packet as a reach check validation. With this change reach checks will not be periodically required for a peer that completes at least one call per 60 seconds. A 1 RTT delay is therefore avoided each time a reach check can be avoided. In addition, reach checks require the service to process an additional ACK packet. Eliminating a large number of reach checks can improve overall service performance.

The final Rx RPC change in this release is specific to kernel implementations. Prior releases restricted the frequency of executing time scheduled Rx events to a granularity no smaller than 500ms. As a result an RTO timer event for a lost packet could not be shorter than 500ms even if the measured RTT for the connection is significantly smaller. The minimum RTO for a connection in AuriStor Rx is 200ms. The inability to schedule shorter timeouts impacts recovery from packet loss.

v2021.05-37 (5 February 2024)

Rx improvements:
- The v2021.05-36 release permanently disabled all use of Rx jumbograms due to a risk of silent data corruption. However, when advertising the number of acceptable datagrams in the ACK trailer a missing htonl() set the value to 16777216 instead of 1 on little-endian systems.
- When sending a PING ACK as a reachability test, ensure that the previousPacket field is properly assigned to the largest accepted DATA packet sequence number instead of zero.
- Replace the initialization state flag with two flags. One that indicates that Rx initialization began and the other that it succeeded. The first prevents multiple attempts at initialization after failure. The second prevents shutdown from accessing uninitialized structures if initialization failed.
Cache Manager Improvements:
- No longer refuse to start if both the 'cachedir' and 'memcache' options are present in the configuration file.
- New variable to store the maximum number of cache blocks used. which is accessible via /proc/fs/auristorfs/cache/blocks_used_max.

v2021.05-36 (10 January 2024)

Rx improvements:
- Permanently disable all use of Rx jumbograms due to a risk of silent data corruption.
  
  Ever since OpenAFS 1.0, and possibly before, a race condition has existed when Rx transmits packets. As the rx_call.lock is dropped when starting packet transmission, there is no protection for data that is being copied into the kernel by sendmsg(). It is critical that this packet data is not modified by another thread. However, races exist between the application, listener, and event threads that can lead to retransmissions starting whilst an original transmission is still in progress. This can lead to the packet headers being overwritten, and either the original transmission, the retransmission or both sending corrupt data to the peer.
  
  This corruption can affect the packet serial number or packet flags. It is particularly harmful when the packet flags are corrupted, as this can lead to multiple Rx packets which were intended to be sent as Rx jumbograms being delivered and misinterpreted as a single large packet. The eventual result of this depends on the Rx security class in play, but it can cause decrypt integrity errors (rxgk:crypt and rxgk:auth) or corruption of the data stream (rxnull, rxgk:clear or rxkad:auth).
  
  All AuriStorFS servers, OpenAFS 1.6 or later servers, and the Windows cache manager have been shipped with Rx jumbograms disabled by default. The UNIX cache managers however are shipped with jumbograms enabled. There are many AFS cells around the world that continue to deploy OpenAFS 1.4 or earlier fileservers which continue to negotiate the use of Rx jumbograms.
  
  It is worth noting that all AuriStorFS v0.198 and later fileservers and cache managers implement explicit checks that will recognize the corrupted application data stream and prevent corrupted file content from being stored either into an AFS vnode's backing store in the volume's object store or the cache manager's AFS cache. OpenAFS cache managers and fileservers do not have these protections.
  
  With Rx jumbograms disabled the maximum number of Rx packets in a datagram is reduced from 6 to 1; the maximum number of send and receive datagram fragments is reduced from 4 to 1; and the maximum advertised MTU is restricted to 1444 - the maximum rx packet size prior to the introduction of jumbograms in IBM AFS 3.5.
- If the rx call flow state transitions from either the RECOVERY or RESCUE states to the LOSS state as a result of an RTO resend event while writing packets to the network, cease transmission of any new DATA packets if there are packets in the resend queue.
- When the call flow state is LOSS and all packets in the resend queue have been retransmitted and yet the recovery point has not been reached, then permit new DATA packets to be sent in order to maintain a full congestion window.
- Add a safety check to prevent the estimated RTT from underflowing when the actual roundtrip time is smaller than 125us.
- Fix the computation of the padding required for rxgk encrypted packets. This bug results in packets sending 8 bytes fewer per packets than the network permits. This bug accidentally prevented the construction of Rx jumbograms when a call is protected by rxgk:crypt.
- Replace the random number generator with a more security source of random bytes.

v2021.05-33 (27 November 2023)

Rx improvements:
- Not all calls transfer enough data to be able to measure a smoothed round-trip time (SRTT). Calls which are unable to compute a SRTT should not be used to update the peer host RTO value which is used to initialize the RTO for subsequent calls.
  
  Without this change, a single DATA packet call will cause the peer host RTO to be reduced to 0ms. Subsequent calls will start with a RTO value of MAX(0, rxi_minPeerTimeout) where rxi_minPeerTimeout defaults to 200ms. If the actual measured RTO is greater than 200ms, then initial RTO will be too small resulting in premature triggering of the RTO timer and the call flow state entering the loss phase which can significantly hurt performance.
- Initialize the peer host RTO to rxi_minPeerTimeout (which defaults to 200ms) instead of one second. Although RFC6298 recommends the use of one second when no SRTT is available, Rx has long used the rxi_minPeerTimeout value for other purposes which are supposed to be consistent with initial RTO value. It should be noted that Linux TCP uses 200ms instead of one second for this purpose.
- If associating a security class with an Rx connection fails immediately place the Rx connection into an error state. A failure might occur if the security class is unable to access valid key material.
- If an incoming Rx call requires authentication and the security class is unable to successfully generate a challenge, put the incoming Rx connection into an error state and issue an abort to the caller.
- If an incoming Rx call requires authentication and the security class is able to generate a challenge but the challenge cannot be returned to Rx, then treat this as a transient error. Do not acknowledge the incoming DATA packet and do not place the Rx connection into an error state. An attempt to re-issue the challenge will be performed when the DATA packet is retransmitted.
- If an Rx call is terminated due to the expiration of the configured connection dead time, idle dead time, hard dead time, or as a result of clock drift, then send an ABORT to the peer notifying them that the call has been terminated. This is particularly important for terminated outgoing calls. If the peer does not know to terminate the call, then the call channel might be in use when the next outgoing call is issued using the same call channel. If the next incoming call is received by an in-use call channel, the receiver must drop the received DATA packet and return a BUSY packet. The call initiator will need to wait for a retransmission timeout to pass before retransmitting the DATA packet. Receipt of BUSY packets cannot be used to keep a call alive and therefore the requested call is at greater risk of timing out if the network path is congested.
aklog and krb5.log (via libyfs_acquire):
- If the linked Kerberos library implements krb5_cc_cache_match() and libacquire has been told to use an explicit principal name and credential cache, the Kerberos library might return KRB5_CC_NOTFOUND even though the requested credential cache is the correct one to use. This release will not call krb5_cc_cache_match() if the requested credential cache contains the requested principal.
Cell Service Database (cellservdb.conf):
- cellservdb.conf has been synchronized with the 31 Oct 2023 update to the grand.central.org CellServDB file.

v2021.05-32 (9 October 2023)

No significant changes for macOS compared to v2021.05-31

v2021.05-31 (25 September 2023)

New platform:
- macOS 14 Sonoma
macOS 14 Sonoma:

AuriStorFS v2021.05-29 and later installers for macOS 13 Ventura are compatible with macOS 14 Sonoma and do not need to be removed before upgrading to macOS 14 Sonoma. Installation of the macOS 14 Sonoma version of AuriStorFS is recommended.

Cache Manager:
- If an AuriStorFS cache manager is unable to use the yfs-rxgk security class when communicating with an AuriStorFS fileserver, it must assume it is IBM AFS 3.6 or OpenAFS and upgrade it to AuriStorFS if an upgrade probe returns a positive result. Once a fileserver's type is identified as AuriStorFS the type should never be reset; even if communication with the fileserver is lost or the fileserver restarts.
  
  If an AuriStorFS fileserver is replaced by an OpenAFS fileserver on the same endpoint, then the UUID of the OpenAFS must be different. As a result, the OpenAFS fileserver will be observed as distinct from the AuriStorFS fileserver that previously shared the endpoint.
  
  Prior to this release there were circumstances in which the cache manager discarded the fileserver type information and would fail to recognize the fileserver as an AuriStorFS fileserver when yfs-rxgk could not be used. This release prevents the cache manager from resetting the type information if the fileserver is marked down.
- If a fileserver's location service entry is updated with a new uniquifier value (aka version number), this indicates that one of the following might have changed:
  1. the fileserver's capabilities
  2. the fileserver's security policy
  3. the fileserver's knowledge of the cell-wide yfs-rxgk key
  4. the fileserver's endpoints
  Beginning with this release the cache manager will force the establishment of new Rx connections to the fileserver when the uniquifier changes. This ensures that the cache manager will attempt to fetch new per-fileserver yfs-rxgk tokens from the cell's RXGK service, enforce the latest security policy, and not end up in a situation where its existing tokens cannot be used to communicate with the fileserver.
aklog:
- Fix incorrect output when populating the server list for a service fails. The stashed extended error explaining the cause of the failure was not displayed.
- If a cell has neither _afs3-prserver._udp. DNS SRV records nor AFSDB records, the lookup of the cell's protection servers would fail if there is no local cell configuration details. The fallback to use _afs3-vlserver._udp. DNS SRV records did not work. This is corrected in this release.

v2021.05-30 (6 September 2023)

Do not mark a fileserver down in response to a KRB5 error code.
fs cleanacl must not store back to the file server a cleaned acl if it was inherited from a directory. Doing so will create a file acl.
Correct the generation of never expire rxkad_krb5 tokens from Kerberos v5 tickets which must have a start time of Unix epoch and an end time of 0xFFFFFFFF seconds. The incorrectly generated tokens were subject to the maximum lifetime of 30 days.
Correct the generation of the yfs-rxgk RESPONSE packet header which failed to specify the key version generation number used to encrypt the authenticator. If the actual key version is greater than zero, then the authenticator would fail to verify.
Enforce a maximum NAT ping period of 20s to ensure that NAT/PAT/firewall rules due not expire while Rx RPCs are in-flight.

v2021.05-29 (26 June 2023)

Execution of fs commands such as examine, whereis, listquota, fetchacl, cleanacl, storeacl, whoami, lsmount, bypassthreshold and getserverprefs could result in memory leaks by the AuriStorFS kernel extension.

v2021.05-27 (1 May 2023)

Fixes for bugs in vos introduced in v2021.05-26.

v2021.05-26 (17 April 2023)

Fixed a potential kernel memory leak when triggered by fs examine, fs listquota, or fs quota.
Increased logging of VBUSY, VOFFLINE, VSALVAGE, and RX_RESTARTING error responses. A log message is now generated whenever a task begins to wait as a result of one of these error responses from a fileserver. Previously, a message was only logged if the volume location information was expired or discarded.
Several changes to optimize internal volume lookups.
Faster failover to replica sites when a fileserver returns RX_RESTARTING, VNOVOL or VMOVED.
rxdebug regains the ability to report rx call flags and rx_connection flags.
The RXRPC library now terminates calls in the QUEUED state when an ABORT packet is received. This clears the call channel making it available to accept another call and reduces the work load on the worker thread pool.
Fileserver endpoint registration changes no longer result in local invalidation of callbacks from that server.
Receipt of an RXAFSCB_InitCallBackState3 RPC from a fileserver no longer resets the volume site status information for all volumes on all servers.

v2021.05-25 (28 December 2022)

The v2021.05-25 release includes further changes to RXRPC to improve reliability. The changes in this release prevent improper packet size growth. Packet size growth should never occur when a call is attempting to recover from packet loss; and is unsafe when the network path's maximum transmission unit is unknown. Packet size growth with be re-enabled in a future AuriStorFS release that includes Path MTU detection and the Extended SACK functionality.
Improved error text describing the source of invalid values in /etc/yfs/yfs-client.conf or included files and directories.

v2021.05-24 (25 October 2022)

New Platform: macOS 13 (Ventura)

v2021.05-22 (12 September 2022) and v2021.05-21 (6 September 2022)

RX RPC
- If receipt of a DATA packet causes an RX call to enter an error state, do not send the ACK of the DATA packet following the ABORT packet. Only send the ABORT packet.
- AuriStor RX has failed to count and report the number of RX BUSY packets that have been sent. Beginning with this change the sent RX BUSY packet count is once again included in the statistics retrieved via rxdebug server port -rxstats.
- Introduce minimum and maximum bounds checks on the ACK packet trailer fields. If the advertised values are out of bounds for the receiving RX stack, do not abort the call but adjust the values to be consistent with the local RX RPC implementation limits. These changes are necessary to handle broken RX RPC implementations or prevent manipulation by attackers.

New to v2021.05-20 (15 August 2022) and v2021.05-19 (13 August 2022)

RX RPC
- Include the DATA packet serial number in the transmitted reachability check PING ACK. This permits the reachability test ACK to be used for RTT measurement.
- Do not terminate a call due to an idle dead timeout if there is data pending in the receive queue when the timeout period expires. Instead deliver the received data to the application. This change prevents idle dead timeouts on slow lossy network paths.
- Fix assignment of RX DATA, CHALLENGE, and RESPONSE packet serial numbers in macOS (KERNEL). Due to a mistake in the implementation of atomic_add_and_read the wrong serial numbers were assigned to outgoing packets.

New to v2021.05-18 (12 July 2022)

Cache Manager
- Prevent a kernel memory leak of less than 64 bytes for each bulkstat RPC issued to a fileserver. Bulkstat RPCs can be frequently issued and over time this small leak can consume a large amount of kernel memory. Leak introduced in AuriStorFS v0.196.
- The Perl::AFS module directly executes pioctls via the OpenAFS compatibility pioctl interface instead of the AuriStorFS pioctl interface. When Perl::AFS is used to store an access control list (ACL), the deprecated RXAFS_StoreACL RPC would be used in place of the newer RXAFS_StoreACL2 or RXYFS_StoreOpaqueACL2 RPCs. This release alters the behavior of the cache manager to use the newer RPCs if available on the fileserver and fallback to the deprecated RPC. The use of the deprecated RPC was restricted to use of the OpenAFS pioctl interface.
RX RPC
- Handle a race during RX connection pool probes that could have resulted in the wrong RX Service ID being returned for a contacted service. Failure to identify that correct service id can result in a degradation of service.
- The Path MTU detection logic sends padded PING ACK packets and requests a PING_RESPONSE ACK be sent if received. This permits the sender of the PING to probe the maximum transmission unit of the path. Under some circumstances attempts were made to send negative padding which resulted in a failure when sending the PING ACK. As a result, the Path MTU could not be measured. This release prevents the use of negative padding.
Preparation for supporting macOS 13 Ventura when it is released in Fall 2022.
Some shells append a slash to an expanded directory name in response to tab completion. These trailing slashes interfered with "fs lsmount", "fs flushmount" and "fs removeacl" processing. This release includes a change to prevent these commands from breaking when presented a trailing slash.

New to v2021.05-17 (16 May 2022)

Cell Service Database Updates
- Update cern.ch, ics.muni.cz, ifh.de, cs.cmu.edu, qatar.cmu.edu, it.kth.se
- Remove uni-hohenheim.de, rz-uni-jena.de, mathematik.uni-stuttgart.de, stud.mathematik.uni-stuttgart.de, wam.umd.edu
- Add ee.cooper.edu
- Restore ams.cern.ch, md.kth.se, italia
Fix parsing of [afsd] rxwindow configuration which can be used to specified a non-default send/receive RX window size. The current default is 128 packets.
RX Updates
- Add nPacketsReflected

Logging improvements

Cache directory validation errors log messages now include the cache directory path.

Log the active configuration path if "debug" logging is enabled.

More details of rxgk token extraction failures.

New to v2021.05-16 (24 March 2022)

RX - Previous releases re-armed the Retransmission Timeout (RTO) each time a new unacknowledged packet is acknowledged instead of when a new leading edge packet is acknowledged. If leading edge data packet and its retransmission are lost, the call can remain in the "recovery" state where it continues to send new data packets until one of the following is true:
. the maximum window size is reached
. the number of lost and resent packets equals 'cwind'
at which point there is nothing left to transmit. The leading edge data packet can only be retransmitted when entering the "loss" state but since the RTO is reset with each acknowledged packet the call stalls for one RTO period after the last transmitted data packet is acknowledged.

This poor behavior is less noticiable with small window sizes and short lived calls. However, as window sizes and round-trip times increase the impact of a twice lost packet becomes significant.

RX - Never set the high-order bit of the Connection Epoch field. RX peers starting with IBM AFS 3.1b through AuriStor RX v0.191 ignore the source endpoint when matching incoming packets to RX connections if the high-order epoch bit is set. Ignoring the source endpoint is problematic because it can result in a call entering a zombie state whereby all PING ACK packets are immediately responded to the source endpoint of the PING ACK but any delayed ACK or DATA packets are sent to the endpoint bound to the RX connection. An RX client that moves from one network to another or which has a NAT|PAT device between it and the service can find themselves stuck.

Starting with AuriStor RX v0.192 the high-order bit is ignored by AuriStor RX peer when receiving packets. This change to always clear the bit prevents IBM AFS and OpenAFS peers from ignoring the source endpoint.

RX - The initial packetSize calculation for a call is altered to require that all constructed packets before the receipt of the first ACK packet are eligible for use in jumbograms if and only if the local RX stack has jumbograms enabled and the maximum MTU is large enough. By default jumbograms are disabled for all AuriStorFS services. This change will have a beneficial impact if jumbograms are enabled via configuration; or when testing RX performance with "rxperf".

New fs whereis -noresolve option displays the fileservers by network endpoint instead of DNS PTR record hostname.

New to v2021.05-15 (24 January 2022)

kernel - fixed YFS_RXGK service rx connection pool leak

New to v2021.05-14 (20 January 2022)

fs mkmount permit mount point target strings longer than 63 characters.
afsd enhance logging of yfs-rxgk token renewal errors.

afsd gains a "principal = configuration option for use with keytab acquisition of yfs-rxgk tokens for the cache manager identity.

kernel - Avoid unnecessary rx connection replacement by racing threads after token replacement or expiration.

kernel - Fix a regression introduced in v2021.05 where an anonymous combined identity yfs-rxgk token would be replaced after three minutes resulting in the connection switching from yfs-rxgk to rxnull.

kernel - Fix a regression introduced in v0.208 which prevented the invalidation of cached access rights in response to a fileserver callback rpc. The cache would be updated after the first FetchStatus rpc after invalidation.

kernel - Reset combined identity yfs-rxgk tokens when the system token is replaced.

kernel - The replacement of rx connection bundles in the cache manager to permit more than four simultaneous rx calls per uid/pag with trunked rx connections introduced the following regressions in v2021.05.

a memory leak of discarded rx connection objects

failure of NAT ping probes after replacement of an connection

inappropriate use of rx connections after a service upgrade failure

All of these regressions are fixed in patch 14.

New to v2021.05-12 (7 October 2021)

fs ignorelist -type afsmountdir in prior releases could prevent access to /afs.

Location server rpc timeout restored to two minutes instead of twenty minutes.

Location server reachability probe timeout restored to six seconds instead of fifty seconds.

Cell location server upcall results are now cached for fifteen seconds.

Multiple kernel threads waiting for updated cell location server reachability probes now share the results of a single probe.

RX RPC implementation lock hierarchy modified to prevent a lock inversion.

RX RPC client connection reference count leak fixed.

RX RPC deadlock during failed connection service upgrade attempt fixed..

New to v2021.05-9 (25 October 2021)

First public release for macOS 12 Monterey build using XCode 13. When upgrading macOS to Monterey from earlier macOS releases, please upgrade AuriStorFS to v2021.05-9 on the starting macOS release, upgrade to Monterey and then install the Monterey specific v2021.05-9 release.

Improved logging of "afsd" shutdown when "debug" mode is enabled.

Minor RX network stack improvements

New to v2021.05-3 (10 June 2021)

Fix for [cells] cellname = {...} without server list.

New to v2021.05 (31 May 2021)

Multi-homed location servers are finally managed as a single server instead of treating each endpoint as a separate server. The new functionality is a part of the wholesale replacement of the former cell management infrastructure. Location server communication is now entirely managed as a cluster of multi-homed servers for each cell. The new infrastructure does not rely upon the global lock for thread safety.

This release introduces a new infrastructure for managing user/pag entities and tracking their per cell tokens and related connection pools.

Expired tokens are no longer immediately deleted so that its possible for them to be listed by "tokens" for up to two hours.

Prevent a lock inversion introduced in v0.208 that can result in a deadlock involving the GLOCK and the rx call.lock. The deadlock can occur if a cell's list of location servers expires and during the rebuild an rx abort is issued.

Add support for rxkad "auth" mode rx connections in addition to "clear" and "crypt". "auth" mode provides integrity protection without privacy.

Add support for yfs-rxgk "clear" and "auth" rx connection modes.

Do not leak a directory buffer page reference when populating a directory page fails.

Re-initialize state when populating a disk cache entry using the fast path fails and a retry is performed using the slow path. If the data version changes between the attempts it is possible for truncated disk cache data to be treated as valid.

Log warnings if a directory lookup operation fails with an EIO error. An EIO error indicates that an invalid directory header, page header, or directory entry was found.

Do not overwrite RX errors with local errors during Direct-I/O and StoreMini operations. Doing so can result in loss of VBUSY, VOFFLINE, UAENOSPC, and similar errors.

Correct a direct i/o code path which could overwrite a fileserver returned error code with a local RXGEN_CC_UNMARSHAL error.

Correct the StoreMini code path which could overwrite a fileserver returned error code with a local RXGEN_CC_UNMARSHAL error.

Ensure the rx call object is not locked when writing to the network socket.

Removed all knowledge of the KERNEL global lock from RX. Acquiring the GLOCK from RX is never safe if any other lock is held. Doing so is a lock order violation that can result in deadlocks.

Fixed a race in the opr_reservation system that could produce a cache entry reference undercount.

If a directory hash chain contains a circular link, a buffer page reference could be leaked for each traversal.

Each AFS3 directory header and page header contains a magic tag value that can be used in a consistency check but was not previously checked before use of each header. If the header memory is zero filled during a lookup, the search would fail producing an ENOENT error. Starting with this release the magic tag values are validated on each use. An EIO error is returned if there is a tag mismatch.

"fs setcrypt -crypt auth" is now a permitted value. The "auth" mode provides integrity protection but no privacy protection.

Add new "aklog -levels " option which permits requesting "clear" and "auth" modes for use with yfs-rxgk.

Update MKShim to Apple OpenSource MITKerberosShim-79.

Report KLL errors via a notification instead of throwing an exception which (if not caught) will result in process termination.

If an exception occurs while executing "unlog" catch it and ignore it. Otherwise, the process will terminate.

New to v2021.04 (22 April 2021)

Primarily bug fixes for issues that have been present for years.

A possibility of an infinite kernel loop if a rare file write / truncate pattern occurs.

A bug in silly rename handling that can prevent cache manager initiated garbage collection of vnodes.

New to v0.209 (13 March 2021)

fs setserverprefs and fs getserverprefs updated to support IPv6 and CIDR specifications.

Improved error handling during fetch data and store data operations.

Prevents a race between two vfs operations on the same directory which can result in caching of out of date directory contents.

Use cached mount point target information instead of evaluating the mount point's target upon each access.

Avoid rare data cache thrashing condition.

Prevent infinite loop if a disk cache error occurs after the first page in a chunk is written.

Network errors are supposed to be returned to userspace as ETIMEDOUT. Previously some were returned as EIO.

When authentication tokens expire, reissue the fileserver request anonymously. If the anonymous user does not have permission either EACCES or EPERM will be returned as the error to userspace. Previously the vfs request would fail with an RXKADEXPIRED or RXGKEXPIRED error.

If growth of an existing connection vector fails, wait on a call slot in a previously created connection instead of failing the vfs request.

Volume and fileserver location query infrastructure has been replaced with a new modern implementation.

Replace the cache manager's token management infrastructure with a new modern implementation.

New to v0.206 (12 January 2021) - Bug fixes

Prevents a possible panic during unmount of /afs.

Improved failover and retry logic for offline volumes.

New to v0.205 (24 December 2020) - Bug fixes

Volume name-to-id cache improvements

Fix expiration of name-to-id cache entries

Control volume name-to-id via sysctl

Query volume name-to-id statistics via sysctl

Improve error handling for offline volumes

Fix installer to prevent unnecessary installation of Rosetta 2 on Apple Silicon

New to v0.204 (25 November 2020) - Bug fix for macOS Big Sur

v0.204 prevents a kernel panic on Big Sur when AuriStorFS is stopped and restarted without an operating system reboot.

introduces a volume name-to-id cache independent of the volume location cache.

New to v0.203 (13 November 2020) - Bug fix for macOS

v0.203 prevents a potential kernel panic due to network error.

New to v0.201 (12 November 2020) - Universal Big Sur (11.0) release for Apple Silicon and Intel

v0.201 introduces a new cache manager architecture on all macOS versions except for High Sierra (10.12). The new architecture includes a redesign of:

kernel extension load

kernel extension unload (not available on Big Sur)

/afs mount

/afs unmount

userspace networking

The conversion to userspace networking will have two user visible impacts for end users:
The Apple Firewall as configured by System Preferences -> Security & Privacy -> Firewall is now enforced. The "Automatically allow downloaded signed software to receive incoming connections" includes AuriStorFS.

Observed network throughput is likely to vary compared to previous releases.

On Catalina the "Legacy Kernel Extension" warnings that were displayed after boot with previous releases of AuriStorFS are no longer presented with v0.201.

AuriStorFS /afs access is expected to continue to function when upgrading from Mojave or Catalina to Big Sur. However, as AuriStorFS is built specifically for each macOS release, it is recommended that end users install a Big Sur specific AuriStorFS package.
AuriStorFS on Apple Silicon supports hardware accelerated aes256-cts-hmac-sha1-96 and aes128-cts-hmac-sha1-96 using AuriStor's proprietary implementation.

New to v0.200 (4 November 2020) - Final release for macOS El Capitan (10.11)

The network path between a client and a server often traverses one or more network segments separated by NAT/PAT devices. If a NAT/PAT times out a RPCs endpoint translation mid-call, this can result in an extended delay before failure and the server being marked down, or worse, a call that never terminates and a client that appears to hang until the fileserver is restarted.

This release includes significant changes to the RX stack and the UNIX cache manager to detect such conditions, fail the calls quickly and detect when it is safe to retry the RPC.

NAT/PAT devices that drop endpoint mappings while in use are anti-social and can result in unwanted delays and even data loss. Whenever possible they should be avoided when they can be. That said, the changes in this release are a huge step to making the loss of endpoint mappings tolerable.

Fix segmentation fault of Backgrounder when krb5_get_credentials() fails due to lack of network connectivity.

Fix the "afsd" rxbind option which was ignored if the default port, 7001, is in use by another process on the system.

If a direct i/o StoreData or FetchData RPC failed such that it must be retried, the retried RPC would fail due to an attempt to Fetch or Store the wrong amount of data. This is fixed.

Servers are no longer marked down if RPCs fail with RX_CALL_PEER_RESET, RX_CALL_EXCEEDS_WINDOW, or RX_PROTOCOL_ERROR. RPCs that are safe to retry are retried.

Fixed a race between a call entering error state and call completion that can result in a the call remaining in the DALLY state and the connection channel remaining in use. If this occurs during process or system shutdown it can result in a deadlock.

During shutdown cancel any pending delayed aborts to prevent a potential deadlock. If a deadlock occurs when unloading a kernel module a reboot will be required.

Updated cellservdb.conf

New to v0.197.1 (31 August 2020) and v0.198 (10 October 2020)

Prevent Dead vnode has core/unlinkedel/flock panic introduced in v0.197.

New to v0.197 (26 August 2020)

A new callback management framework for UNIX cache managers reduces the expense of processing volume callback RPCs from O(number of vcache objects) to O(1). A significant amount of lock contention has been avoided. The new design reduces the risk of the single callback service worker thread blocking. Delays in processing callbacks on a client can adversely impact fileserver performance and other clients in the cell.

Bulk fetch status RPCs are available on macOS for the first time. Bulk fetch status permits optimistic caching of vnode status information without additional round-trips. Individual fetch status RPCs are no longer issued if a bulk status fails to obtain the required status information.

Hardware accelerated crypto is now available for macOS cache managers. AuriStor's proprietary aes256-cts-hmac-sha1-96 and aes256-cts-hmac-sha512-384 implementations leverage Intel processor extensions: AESNI AVX2 AVX SSE41 SSSE3 to achieve the fastest encrypt, decrypt, sign and verify times for RX packets.

This release optimizes the removal of "._" files that are used to store extended attributes by avoiding unnecessary status fetches when the directory entry is going to be removed.

When removing the final directory entry for an in-use vnode, the directory entry must be silly renamed on the fileserver to prevent removal of the backing vnode. The prior implementation risked blindly renaming over an existing silly rename directory entry.

Behavior change! When the vfs performs a lookup on ".", immediately return the current vnode.

if the object is a mount point, do not perform fakestat and attempt to resolve the target volume root vnode.

do not perform any additional access checks on the vnode. If the caller already knows the vnode the access checks were performed earlier. If the access rights have changed, they will be enforced when the vnode is used just as they would have if the lookup of "." was performed within the vfs.

do not perform a fetch status or fetch data rpcs. Again, the same as if the lookup of "." was performed within the vfs.

Volumes mounted at more than one location in the /afs namespace are problematic on more than one operating system that do not expect directories to have more than one parent. It is particularly problematic if a volume is mounted within itself. Starting with this release any attempt to traverse a mountpoint to the volume containing the mountpoint will fail with ENODEV.

When evaluating volume root vnodes, ensure that the vnode's parent is set to the parent directory of the traversed mountpoint and not the mountpoint. Vnodes without a parent can cause spurious ENOENT errors on Mojave and later.

v0.196 was not publicly released.

New to v0.195 (14 May 2020)

This is a CRITICAL update for AuriStorFS macOS clients.

In Sep 2019 AuriStorFS v0.189 was released which provided faster and less CPU intensive writing of (>64GB) large files to /afs. These improvements introduced a hash collision bug in the store data path of the UNIX cache manager which can result in file corruption. If a hash collision occurs between two or more files that are actively being written to via cached I/O (not direct I/O), dirty data can be discarded from the auristorfs cache before it is written to the fileserver creating a file with a range of zeros (a hole) on the fileserver. This hole might not be visible to the application that wrote the data because the lost data was cached by the operating system. This bug has been fixed in v0.195 and it is for this reason that v0.195 has been designated a CRITICAL release for UNIX/Linux clients.

While debugging a Linux SIGBUS issue, it was observed that receipt of an ICMP network error in response to a transmitted packet could result in termination of an unrelated rx call and could mark a server down. If the terminated call is a StoreData RPC, permanent data loss will occur. All Linux clients derived from the IBM AFS code base experience this bug. The v0.195 release prevents this behavior.

This release includes changes that impact all supported UNIX/Linux cache managers. On macOS there is reduced lock contention between kernel threads when the vcache limit has been reached.

The directory name lookup cache (DNLC) implementation was replaced. The new implementation avoids the use of vcache pointers which did not have associated reference counts, and eliminates the invalidation overhead during callback processing. The DNLC now supports arbitrary directory name lengths; the prior implementation only cached entries with names not exceeding 31 characters.

Prevent matching arbitrary cell name prefixes as aliases. For example "/afs/y" should not be an alias for "your-file-system.com". Some shells, for example "zsh", query the filesystem for names as users type. Delays between typed characters result in filesystem lookups. When this occurs in the /afs dynroot directory, this could result in cellname prefix string matches and the dynamic creation of directory entries for those prefixes.

New to v0.194 (2 April 2020)

This is a CRITICAL release for all macOS users. All prior macOS clients whether AuriStorFS or OpenAFS included a bug that could result in data corruption either when reading or writing.

This release also fixes these other issues:

sign and notarize installer plugin "afscell" bundle. The lack of digital signature prevented the installer from prompting for a cellname on some macOS versions.

prevent potential for corruption when caching locally modified directories.

v0.193 was withdrawn due to a newly introduced bug that could result in data corruption.

New to v0.192 (30 January 2020)

The changes improve stability, efficiency, and scalability. Post-0.189 changes exposed race conditions and reference count errors which can lead to a system panic or deadlock. In addition to addressing these deficiencies this release removes bottlenecks that restricted the number of simultaneous vfs operations that could be processed by the AuriStorFS cache manager. The changes in this release have been successfully tested with greater than 400 simultaneous requests sustained for for several days.

New to v0.191 (16 December 2019)

Restore keyed cache manager capability broken in v0.189.

Add kernel module version string to AuriStorFS Preference Pane.

Other kernel module bug fixes.

New to v0.190 (14 November 2019)

Short-circuit busy volume retries after volume or volume location entry is removed.

New to v0.189 (28 October 2019)

Faster "git status" operation on repositories stored in /afs.

Faster and less CPU intensive writing of (>64GB) large files to /afs. Prior to this release writing files larger than 1TB might not complete. With this release store data throughput is consistent regardless of file size. (See "UNIX Cache Manager large file performance improvements" later in this file).

macOS Catalina (8 October 2019)

AuriStorFS v0.188 released for macOS Catalina (10.15)

New to v0.188 (23 June 2019)

Increased clock resolution for timed waits from 1s to 1ns

Added error handling for rx multi rpcs interrupted by signals

New to v0.186 (29 May 2019)

v0.184 moved the /etc/yfs/cmstate.dat file to /var/yfs. With this change afsd would fail to start if /etc/yfs/cmstate.dat exists but contains invalid state information. This is fixed.

v0.184 introduced a potential deadlock during directory processing. This is fixed.

Handle common error table errors obtained outside an afs_Analyze loop. Map VL errors to ENODEV and RX, RXKAD, RXGK errors to ETIMEDOUT

Log all server down and server up events. Transition events from server probes failed to log messages.

RX RPC networking:

If the RPC initiator successfully completes a call without consuming all of the response data fail the call by sending an RX_PROTOCOL_ERROR ABORT to the acceptor and returning a new error, RX_CALL_PREMATURE_END, to the initiator.
Prior to this change failure to consume all of the response data would be silently ignored by the initiator and the acceptor might resend the unconsumed data until any idle timeout expired. The default idle timeout is 60 seconds.

Avoid transmitting ABORT, CHALLENGE, and RESPONSE packets with an uninitialized sequence number. The sequence number is ignored for these packets but set it to zero.

New to v0.184 (26 March 2019)

The initial congestion window has been reduced from 10 Rx packets to 4. Packet reordering and loss has been observed when sending 10 Rx packets via sendmmsg() in a single burst. The lack of udp packet pacing can also increase the likelihood of transmission stalls due to ack clock variation.

The UNIX Cache Manager underwent major revisions to improve the end user experience by revealing more error codes, improving directory cache efficiency, and overall resiliency. The cache manager implementation was redesigned to be more compatible with operating systems such as Linux and macOS that support restartable system calls. With these changes errors such as "Operation not permitted", "No space left on device", "Quota exceeded", and "Interrupted system call" can be reliably reported to applications. Previously such errors might have been converted to "I/O error".

New to v0.180 (9 November 2018)

RX reliability and performance improvements for high latency and/or lossy network paths such as public wide area networks.

A fix for a macOS firewall triggered kernel panic introduced in v0.177.

New to v0.177 (17 October 2018)

A fix to AuriStor's RX implementation bug introduced in v0.176 that interferes with communication with OpenAFS and IBM Location and File Services.

New to v0.176 (3 October 2018)

AuriStor's RX implementation has undergone a major upgrade of its flow control model. Prior implementations were based on TCP Reno Congestion Control as documented in RFC5681; and SACK behavior that was loosely modelled on RFC2018. The new RX state machine implements SACK based loss recovery as documented in RFC6675, with elements of New Reno from RFC5682 on top of TCP-style congestion control elements as documented in RFC5681. The new RX also implements RFC2861 style congestion window validation.

When sending data the RX peer implementing these changes will be more likely to sustain the maximum available throughput while at the same time improving fairness towards competing network data flows. The improved estimation of available pipe capacity permits an increase in the default maximum window size from 60 packets (84.6 KB) to 128 packets (180.5 KB). The larger window size increases the per call theoretical maximum throughput on a 1ms RTT link from 693 mbit/sec to 1478 mbit/sec and on a 30ms RTT link from 23.1 mbit/sec to 49.39 mbit/sec.

Improve shutdown performance by refusing to give up callbacks to known unreachable file servers and apply a shorter timeout period for the rest.

Permit RXAFSCB_WhoAreYou to be successfully executed after an IBM AFS or OpenAFS fileserver unintentionally requests an RX service upgrade from RXAFSCB to RXYFSCB.

RXAFS timestamps are conveyed in unsigned 32-bit integers with a valid range of 1 Jan 1970 (Unix Epoch) through 07 Feb 2106. UNIX kernel timestamps are stored in 32-bit signed integers with a valid range of 13 Dec 1901 through 19 Jan 2038. This discrepency causes RXAFS timestamps within the 2038-2106 range to display as pre-Epoch.

RX Connection lifecycle management was susceptible to a number of race conditions that could result in assertion failures, the lack of a NAT ping connection to each file server, and the potential reuse of RX connections that should have been discarded.

This release includes a redesigned lifecycle that is thread safe, avoids assertions, prevents NAT ping connection loss, and ensures that discarded connections are not reused.

The 0.174 release unintentionally altered the data structure returned to xstat_cm queries. This release restores the correct wire format.

Since v0.171, if a FetchData RPC fails with a VBUSY error and there is only one reachable fileserver hosting the volume, then the VFS request will immediately with an ETIMEDOUT error ("Connection timed out").

v0.176 corrects three bugs that contributed to this failure condition. One was introduced in v0.171, another in 0.162 and the final one dates to IBM AFS 3.5p1.

The intended behavior is that a cache manager, when all volume sites fail an RPC with a VBUSY error, will sleep for up to 15 seconds and then retry the RPC as if the VBUSY error had never been received. If the RPC continues to receive VBUSY errors from all sites after 100 cycles, the request will be failed with EWOULDBLOCK ("Operation would block") and not ETIMEDOUT.

Prefer VOLMISSING and VOLBUSY error states to network error states when generating error codes to return to the VFS layer. This will result in ENODEV ("No such device") errors when all volume sites return VNOVOL or VOFFLINE errors and EWOULDBLOCK ("Operation would block") errors when all volume sites return VBUSY errors. (v0.176)

New to v0.174 (24 September 2018)

macOS Mojave (10.14) support

New to v0.170 (27 April 2018)

Faster processing of cell configuration information by caching service name to port information.

RX call sequence number rollover to permit calls that require the transmission of more than 5.5TB of data.

Command parser Daylight Saving Time bug fix

Fix a bug that prevented immediate access to a mount point created with "fs mkmount" on the same machine.

Fix the setting of "[afsd] sysnames = " during cache manager startup.

New to v0.168 (6 March 2018)

Corrects "fs setacl -negative" processing [CVE-2018-7168]

Improved reliability for keyed cache managers. More persistent key acquisition renewals.

Major refresh to cellservdb.conf contents.

DNS SRV and DNS AFSDB records now take precedence when use_dns = yes

Kerberos realm hinting provided by
kerberos_realm = [REALM]

DNS host names are resolved instead of reliance on hard coded IP addresses
The cache manager now defaults to sparse dynamic root behavior. Only thiscell and those cells that are assigned aliases are included in /afs directory enumeration at startup. Other cells will be dynamically added upon first access.
Several other quality control improvements.

New to v0.167 (7 December 2017)

Addresses a critical remote denial of service vulnerability [CVE-2017-17432]
Alters the volume location information expiration policy to reduce the risk of single points of failures after volume release operations.
'fs setquota' when issued with quota values larger than 2TB will fail against OpenAFS and IBM AFS file servers
Memory management improvements for the memory caches.

New to v0.164 (11 November 2017)

Internal cache manager redesign. No new functionality.

New to v0.160 (21 September 2017)

Support for OSX High Sierra's new Apple File System (APFS). Customers must upgrade to v0.160 or later before upgrading to OSX High Sierra.
Reduced memory requirements for rx listener thread
Avoid triggering a system panic if an AFS local disk cache file is deleted or becomes inaccessible.
Fixes to "fs" command line output

New to v0.159 (7 August 2017)

Improved failover behavior during volume maintenance operations
Corrected a race that could lead the rx listener thread to enter an infinite loop and cease processing incoming packets.

New to v0.157 (12 July 2017)

Bundled with Heimdal 7.4 to address CVE-2017-11103 (Orpheus' Lyre puts Kerberos to sleep!)
"vos" support for volume quotas larger than 2TB.
"fs flushvolume" works
Fixed a bug that can result in a system panic during server capability testing

New to v0.150

AuriStorFS file server detection improvements

New to v0.149

rxkad encryption is enabled by default. Use "fs setcrypt off" to disable encryption when tokens are available.
Fix a bug in atomic operations on Sierra and El Capitan which could adversely impact Rx behavior.

New to v0.128

Extended attribute ._ files are automatically removed when the associated files are unlinked
Throughput improvements when sending data

New to v0.121

OSX Sierra support

New to v0.117

Cache file moved to a persistent location on local disk
AuriStor File System graphics
Improvements in Background token fetch functionality
Fixed a bug introduced in v0.44 that could result in an operating system crash when enumerating AFS directories containing Unicode file names (v0.106)
El Capitan security changes prevented Finder from deleting files and directories. As of v0.106, the AuriStor OSX client implements the required functionality to permit the DesktopHelperService to securely access the AFS cache as the user permitting Finder to delete files and directories.

Features:

Not vulnerable to OPENAFS-SA-2015-007.
Office 2011 can save to /afs.
Office 2016 can now save files to /afs.
OSX Finder and Preview can open executable documents without triggering a "Corrupted File" warning. .AI, .PDF, .TIFF, .JPG, .DOCX, .XLSX, .PPTX, and other structured documents that might contain scripts were impacted.
All file names are now stored to the file server using Unicode UTF-8 Normalization Form C which is compatible with Microsoft Windows.
All file names are converted to Unicode UTF-8 Normalization Form D for processing by OSX applications.

Known issues:

None

macOS Installer (10.15 Catalina)

Release Notes

Known Issues

If the Kerberos default realm is not configured, a delay of 6m 59s can occur before the AuriStorFS Backgrounder will acquire tokens and display its icon in the macOS menu. This is the result of macOS performing a Bonjour (MDNS) query in an attempt to discover the local realm.
"aklog" will fail to acquire tokens if the current working directory is located in an inaccessible AFS volume. When the "cwd" is inaccessible, the Heimdal framework is unable to initialize the Kerberos v5 context and reports an out of memory error.

New v2021.05-64 (22 Aug 2025)

The v2021.05-64 release fixes a race condition which can lead to a kernel panic when executing the "unlog" command.

v2021.05-63 (31 May 2025)

When AuriStorFS cache managers added support for per-file ACLs in v0.1, the masking of access rights based upon the inode's S_IRWXU mode bits were skipped when reading objects from AuriStorFS fileservers. The masking continued to be applied when reading objects from OpenAFS fileservers. This release applies S_IRWXU mode bits masking of granted access rights to both AuriStorFS and OpenAFS fileservers.
The existing fake stat mountpoint evaluation logic assumes that the fileserver returned a AFSFetchStatus.ParentVnode that refers to the parent directory from which the mountpoint was located. However, this is only guaranteed to be true if the mountpoint does not have a per-file ACL (which is permitted) and has never been cross-directory hard linked (which is also permitted).
This release improves the error handling logic to fail the fake stat evaluation if the AFSFetchStatus.ParentVnode does not match expectations.
This release fixes a metadata inconsistency when a mountpoint is created when disconnected mode is enabled. A mountpoint should be reported to the vfs as a directory and not a symlink.
The cache manager must track for each inode whether the object's status infomration was received from an AFS3 fileserver or an AuriStorFS fileserver. This release alters when that information is recorded to ensure that it is recorded each time the inode's status information is updated by a fileserver response.
This knowledge is required to properly determine how to interpret the status information.
Directory entries referring to mountpoints and volume roots must not be stored within the Directory Name Lookup Cache (DNLC). Instead, full lookups must be performed each time to ensure proper mountpoint resolution.
Prior to this release locally created mountpoints were stored into the DNLC by mistake.

v2021.05-62 (24 April 2025)

macOS Sequoia v15.4 vs Cisco AnyConnect VPN

Beginning with macOS 15.4, use of Cisco AnyConnect VPN breaks DNS SRV, AFSDB and TXT record lookups performed using the cross-platform DNS resolver API. As a result, clients which rely upon DNS-based zero-configuration resolution of Kerberos realms and/or AFS cells database services are unable to access those resources. This release switches the macOS builds to use the Apple DNS API.
The lookup failures impacted all command line tools, clients and services.
Note: Communication failures can occur if the VPN is configured by the organization to block the transmission of fragmented UDP packets. Set [afsd] rxmaxmtu = 1328 to ensure all RX packets are small enough to traverse the VPN link.
This release improves consistency of vnode metadata interpretation when the status information is received via an RPC which does not return a callback.

v2021.05-61 (31 March 2025)

Instead of allocating the same sized hash table for the vcache and dcache objects regardless of how many elements are expected to be stored, scale the hash table sizes. The vcache hash table will scale from 4096 buckets to 131072 buckets with a target of 4 entries per bucket. The dcache hash tables will scale from 4096 buckets to 65536 buckets with a target of 2 entries per bucket.
Historically, afs cache managers which create symlinks (including mountpoints) locally allocate a fake infinite lifetime callback to permit applications to use the symlink (or mountpoint) without requiring FetchStatus RPCs to be issued to the fileserver. This approach has drawbacks because it means that fileservers will not issue a callback break when the volume containing the symlink is taken offline, moved, deleted, etc. It also means that a callback will not be sent if another cache manager removes the symlink.
Beginning with this release the AuriStorFS cache manager will no longer allocate a fake infinite lifetime callback and will instead rely upon fileserver allocated callbacks.
"fs makemount" is supposed to create a mountpoint even if the target volume does not exist or is inaccessible. v2021.05 introduced support for creating mountpoints which kafs which broke the ability to create mountpoints to non-existent volumes.

v2021.05-60 (6 March 2025)

The cache manager retries fileserver RPCs up to 100 times when the fileserver replies with VBUSY, VOFFLINE or VSALVAGE. The cache manager fails the application request after the limit is reached. Prior to this release, the failure would result in the cache manager returning an ENODEV error to the application. But ENODEV indicates that the required volume does not exist and the fileserver has not indicated that the volume does not exist. The cache manager will now return EWOULDBLOCK to the application because the volume might be temporarily inaccessible.
Many OpenAFS and AuriStorFS fileservers can return VNOVOL as a side effect of a race between a volume going offline so that control can be given to the Volume Management Service (volserver) and an incoming RPC requesting that the volume be attached. A VNOVOL can also be returned because the volume was temporarily deleted from the vice partition prior to being replaced by a new clone, or prior to the fileserver marking the volume group as having been moved to another fileserver.
The fileserver replying to an RPC with VNOVOL cannot be trusted to mean that the volume has been permanently removed unless it is matched by the Location Service reporting the Volume Location (VL) entry has been deleted. From now on when a VNOVOL response is received and the VL entry exists, the cache manager will return EWOULDBLOCK instead of ENODEV to the application.
Prior to this release, any VL error code returned from the Location Service would be interpreted as meaning that the requested VL entry does not exist. Starting with this release only the VL_NOENT error will be interpreted as the VL entry does not exist.
Restrict the internal use of the ENOENT error code so the only use of ENOENT is to indicate that a directory entry does not exist. Previously, ENOENT could be returned if a disk cache entry could not be allocated or found. A disk cache related ENOENT returned when processing a directory might have been confused for the requested directory entry not existing.
Restrict the internal use of the ENODEV error code so that the only meaning of ENODEV is that the required volume does not exist. Previously, ENODEV could be returned due to transient errors when failing to contact the Location Service.

v2021.05-59 (13 February 2025)

The fix for garbage collection of idle unixuser objects introduced in v2021.05-56 was incomplete. As a result, Rx client connections might not be garbage collected.

v2021.05-57 (9 February 2025)

No new changes for macOS.

v2021.05-56 (7 February 2025)

Minor bug fixes to the Rx RPC implementation
The fix for garbage collection of idle unixuser objects introduced in v2021.05-55 was incomplete. As a result, Rx client connections might not be garbage collected.

v2021.05-55 (23 January 2025)

Minor bug fixes to the Rx RPC implementation
During garbage collection of idle unixuser objects, any Rx client connections associated with the unixuser (uid or pag) that were in use by another thread would not have their cache manager reference dropped. As a result the Rx client connections could never be garbage collected which is problematic on systems which churn through many unixusers. The behavior has been changed so that unixusers objects with in use Rx connections will not be freed. This permits future attempts to garbage collect unixusers to do so only after the Rx connections can also be freed.
Although this misbehavior has been present for decades, it is only since v2021.05-52 that the problem has been observed. It is possible that the changes to reduce the holding of the GLOCK has created a window of opportunity for connections to be simultaneously in use during unixuser garbage collection.

v2021.05-54 (19 January 2025)

The UNIX cache manager changes continue AuriStor's efforts to address a report that was received at the end of September. The end user site uses an OpenAFS cell to distribute software for execution by IBM Spectrum LSF (load sharing facility) managed compute nodes. Each of the approximately one dozen compute nodes is configured with approximately 100 processor cores and 1TB of ram. Each job scheduled by LSF runs in a newly allocated Process Authentication Group (PAG). Some jobs run for hours or days; others complete in a few seconds. LSF has been observed scheduling more than 3000 jobs in under a minute with peak counts of cache manager user objects at 1300. Everything ran well until it didn't. After some unexplained trigger event, all of the remaining free memory on the system would be consumed in about an hour causing the system to panic.

Over the last five months AuriStor has diligently worked to develop theories, optimize the AuriStorFS UNIX cache manager to address each theory; then move on to subsequent ideas once prior theories were disproved. Over this time we have replaced the memory management lifecycle for access rights entries and rx events. We have made it possible for Linux permission checks to be processed without acquiring the GLOCK in the majority of access patterns thereby increasing parallelism. The lifecycle management of PAG allocation and user entity tracking has been rewritten to avoid the GLOCK. The Linux inode lifecycle management has been rewritten which some sites attribute to elimination of swapping on their systems. All of these changes increased throughput and parallelism within and without the cache manager. However, none of them prevented the memory consumption once the unknown trigger event occurred.

It is with great pleasure that we can report that the trigger event and the source of the memory consumption have been identified. The trigger is a damaged Rx packet header which causes the AuriStor Rx stack to place an Rx connection into an RX_PROTOCOL_ERROR error state when there is an in-flight call to an OpenAFS fileserver. The memory consumption is due to the infinite loop that occurs when an Rx connection is in an error state and the error code is not one of the error codes which forces replacement of the Rx connection. Within each pass of the loop a structure is allocated to track the history of per-server errors so they can be logged when hard mount mode is enabled for the cell. AuriStor Rx permits up to 12 calls on each Rx connection. As a result there can be up to a dozen processes each spinning and allocating memory when the Rx connection is placed into the error state.

Days before Christmas 2023 AuriStor identified a race condition which can result in damaged Rx DATA packets being written to the network. Initial transmission of Rx DATA packets can be sent either from an application thread or an Rx listener. Packet retransmission occurs either in an Rx listener thread or in the Rx event thread. The race permits a second thread to modify the Rx packet headers while the sendmsg() syscall from the original transmission is executing. This race is possible because the original thread might exhaust its timeslice during the sendmsg() call and it might be many milliseconds before it is rescheduled on a busy system. In the meantime, packet retransmissions might be initiated either by an Rx listener thread processing an ACK packet or the Rx event thread processing a timeout event. If the initial packet transmission completes while the Rx headers are being rewritten the resulting packet sent on the wire might be garbage. AuriStor observed that if the modified packets were jumbograms that silent data corruption could occur so we disabled all use of jumbograms in v2021.05-35 We also notified both IBM and OpenAFS of the problem.

v2021.05-53 modified the RPC result analyze logic to force the replacement of Rx connections in an error state except when the error signals a token expiration. When a token is expired there is no benefit to replacing the connection until new tokens have been acquired. This release further ensures that infinite memory allocation cannot occur by tracking only the most recent error per fileserver; and only does so if hard mount mode is enabled for the cell.

Processes that issued syscalls which became stuck in a RPC retry loop could not be interrupted. In addition to the unintentional retry loops there are intentional retry loops such as those that occur because a fileserver repeated fails the call with a “VBUSY” error. “VBUSY” originally meant that the required volume was temporarily in use by the volume management service (“volserver”) and would be back in service soon so please wait. Beginning with IBM AFS 3.4 the error code was used to force a retry for reasons having nothing to do with the requested volume’s availability. IBM AFS 3.4 introduced the concept of multi-homed clients which meant that the fileserver could no longer track the identity of a cache manager by its IPv4 address. Instead, each cache manager randomly generates a UUID which the fileserver must query each time an incoming RPC is received on a previously unrecognized Rx connection. An Rx connection might be unrecognized because it’s new, or because the client’s IP address changed, or because the fileserver garbage collected its state information. Regardless of the reason, if a call is received and there is no Rx connection binding to a cache manager UUID, then the fileserver application worker thread to which the call was assigned must issue an outgoing RPC to the client’s RXAFSCB service. Each client has at most one worker thread. Prior to this release, every incoming RXAFSCB call required acquisition of the GLOCK before it could execute. If the GLOCK is unavailable, the RXAFSCB thread blocks and no other pending RXAFSCB calls can be processed. If a RXAFSCB call blocks for too long, the fileserver timeouts trigger and the fileserver replies to the incoming RXAFS call with “VBUSY”. “VBUSY” can also be sent if the fileserver’s quota on the number of calls from the same endpoint is exceeded.

The v2021.05-54 release removes the GLOCK requirement for many RXAFSCB RPCs including all of the UUID query RPCs and all whole volume callback RPCs. Callback RPCs for individual vnodes must still acquire the GLOCK. However, it is hoped that these changes will significantly reduce the time required to process the queries generated due to unrecognized Rx connections. The AuriStor Rx stack permits a dozen calls to be issued on a single logical connection which the fileserver sees as three unique Rx connections from the same endpoint.

Each Unix UID or PAG ID must use an independent Rx connection to each fileserver. On a system which processes /afs syscalls for 1200 UID or PAG IDs in a minute, there is the potential for 3600 Rx connections per fileserver. When the fileserver is OpenAFS, this quickly triggers per endpoint throttling and generates VBUSY storms which can result in unlucky client processes experiencing starvation. With this release, starving processes can be interrupted and RXAFSCB UUID queries can be completed without blocking.

AuriStor believes this release will provide significant benefits for large scale compute nodes, web servers and time sharing systems.

v2021.05-53 (4 January 2025)

v2021.05-52 switched the memory allocation strategy for rx_event objects from a global free list to reliance upon the operating system's memory allocator to allocate individual objects which are freed upon rx_event completion or cancelation. Reliance upon the system allocator introduces the risk of allocation failures due to out of memory conditions. If allocation of an rx_event object fails, a reference count leak on the associated rx_connection or rx_call object can occur preventing garbage collection of the object.
This release handles rx_event allocation failures:
- delayed ACK or delayed ABORT events are replaced by immediate transmission of the ACK or ABORT
- other cases fail the call or connection with an RX_OUT_OF_MEMORY error which is translated on the wire to RX_WIRE_CALL_DEAD (-1).
After executing a call to a fileserver, if the Rx connection is in an error state for any reason other than expired tokens, then schedule the Rx connection to be replaced the next time a call is attempted. All calls issued on an Rx connection that is already in an error state fail immediately preventing further communication with the associated fileserver from the UID/PAG.
Rx connections in an expired token error state are not replaced until new tokens are set.
Rx connections can be placed into an error state either due to local or remote failures. Such failures are often generated by the in-use security class (RXKAD/RXGK) but might be due to an out-of-memory condition, system clock advancement or reversal, or even application errors.
This release includes internal code changes to the management of servers, connections and access rights which will be used by a future release to separate PAG ID allocation from the system's UID namespace.
If 'afsd' is configured with a keytab, receipt of a PoweredOn event will force an attempt to acquire system tokens.
On supported macOS versions, enforce the use of code signing identities to authenticate mach messages involving the AFSBackgrounder and the AFSPreference panel.

v2021.05-52 (5 December 2024)

The v2021.05-50 replacement of the linked list implementation of the vcache access cache with a btree and last lookup cache introduced a memory corruption bug which is fixed in this release.
Eliminate the GLOCK requirement for the collection of function call and RPC statistics collection.
Replace small space allocation with a GLOCK free allocator.
Reimplement the cache manager tracing logging routines so they no longer require the GLOCK.
The Rx networking stack schedules many different types of time sensitive events associated with individual packet retransmission, delayed ack and abort processing, path mtu detection, keep alives, and other categories. Prior to this release, all memory allocated to Rx events objects would not be freed to the operating system until the userspace process terminated or the kernel module was unloaded.
Beginning with this release, instead of having a single, global, free list for rxevents, the system allocator is trusted to be able to allocate memory quickly. This simplifies the Rx implementation, and should be more efficient, as most system allocators will have per thread or core slabs, removing the contention of the global lock protecting the rxevent free list. In addition, the change protects against a network event allocating memory that cannot be readily freed.
On non-Total Store Ordering (TSO)architectures, relaxed atomics used as reference counters do not ensure completion of all prior stores or recency of later loads. This release replaces all read-modify-write atomics to be acquire-release, which, among other things, ensures that only the intended ordering is possible.

v2021.05-51 (26 November 2024)

Fix a vcache (aka inode) reference leak that can occur during an access rights query for a non-directory object if the neither the calling process nor the cache manager possesses tokens for the cell containing the object. The leak was introduced in v2021.05-45.
When scanning the token list for each user searching for expired tokens the user spinlock must be held to prevent races. The updated user management implementation pulled into v2021.05-45 failed to hold the spinlock.
During cache manager module shutdown reset global structures to the module initialization state. Doing so improves the likelihood that the module can be reinitialized.

v2021.05-50 (23 November 2024)

The linked list implementation of the vcache access rights cache was replaced with a btree and a last lookup cache. The btree design is more efficient for lookups and removals, and scales better as the number of uids and/or process authentication groups (pags) accessing /afs increases.
Prune access rights for expired users. Doing so reduces the cost of subsequent lookups.
Cache the access rights obtained for a uid|pag via the "anyAccess" field. Previously, a uid|pag that obtained its rights from the "anyAccess" field had to perform and fail a rights lookup on each and every vcache access.
When setting or clearing tokens for a single cell, no longer invalidate the access rights cached for the uid|pag for all cells. Instead, only invalidate the access rights cached for the cell whose tokens were set or cleared.

v2021.05-49 (16 November 2024)

The output of "tokens" command failed to report yfs-rxgk tokens was broken starting in v2021.05-46

v2021.05-48 (12 November 2024)

Preallocated buffer overflows in XDR responses (CVE-2024-10397)
The AuriStorFS and AFS3 RPC suites rely upon Sun RPC XDR to marshal binary data structures for network transfer. The AuriStor XDR implementation is derived from Sun Microsystems' Sun RPC code base. The Sun RPC XDR API permits memory for output parameters to (optionally) be preallocated which can result in various classes of memory corruption and/or memory leaks in RPC initiator processes.
The AuriStorFS v2021.05-48 release introduces additional data length validation checks within the AuriStor XDR implementation and prohibits the use of preallocated memory for string output parameters or fields. All cache managers, servers and command line tools are modified by these changes.

v2021.05-46 (28 October 2024)

Cache Manager:
- Prevent a kernel memory leak when server preferences are set via the yfs-client.conf [afsd] configuration or via "fs setserverprefs".
- Directory enumeration of a truncated directory now returns an error instead of assuming the end of the directory has been reached.
- Since AFS 3.0, the Unix cache manager has used the root identity credentials to create anonymous outgoing connections to the location service and each fileserver. However, if uid 0 is assigned a token, then those Rx connections will no longer be anonymous. Beginning with this release anonymous outgoing connections are always created with the NOPAG identity (uid 0xffffffff) instead of the root identity.
- When establishing an outgoing rxgk connection, do not fallback to the systemuser's credentials if the user's credentials resulted in a fatal error. Falling back to the systemuser's credentials can result in inappropriate use of an anonymous connection.
- Improved access rights cache correctness for YFS servers
  In prior releases, the access check logic used the file rights for any files fetched from an AuriStorFS fileserver. For files fetched from an AFS-3 fileserver (and, historically, for all files), it used the directory rights, with the (a)dmin right from the file mixed in. The (a)dmin right on a non-directory indicates that the object is owned by the authenticated user.
  This approach has some issues when combined with the access rights cache, and current fileserver callback behavior. On an AuriStorFS file server, the rights on a non-directory may be determined by the rights granted on its parent directory or, with per-file ACLs, those granted on the object itself. The fileserver will only break a non-directory's callback when a per-file ACL is changed - changing a directory ACL will not break callbacks on files within that directory. This means that changing a directory ACL will not invalidate access rights cache entries on files in tha directory, even if the effective ACL on this files has changed, and the cached rights are no longer correct.
  This release works around this by adding a new function which returns the access rights for a file hosted on an AuriStor fileserver. It uses the parent vnode information to locate the parent directory. If the parent directory isn't in the cache, or it doesn't have a valid callback, or if it has been changed since the file's access rights were cached, it clears the current access rights. Files without a parent directory must have per-file ACLs, and so their cached rights can be safely used.
  Note that files with parent vnodes may still have per-file ACLs, and that the breadcrumbing performed by the client may add parent vnode fields to vnodes which don't have them provided by the fileserver. Such vnodes may have their cached access rights cleared more frequently than necessary.
- Add a new mechanism for caching access rights within the vcache structure. This cache is protected via a vcache specific spinlock, and can be accessed without holding the GLOCK.
  This new cache mechanism returns the memory associated with cached rights back to the kernel's slab free memory pool instead of adding the unused rights structures to a cache manager managed free list. The previous cache implementation never returned allocated memory to the kernel. Instead, invalidated access rights were appended to a free access rights queue for later reuse.
- When a volume is accessed via multiple mountpoints a choice must be made regarding which mountpoint is considered to be the active (or parent) mountpoint. This release alters the behavior such that the active mountpoint is set every time a mountpoint is traversed.
  This behavior is easier to understand and is more likely to provide the expected result for a single process that repeatedly accesses volumes from multiple mountpoints. However, it can result in unexpected results when multiple processes are traversing multiple mountpoints in parallel without any synchronization.

v2021.05-45 (Not released)

v2021.05-44a (18 September 2024)

Authentication:
- AuriStorFS v2021.05-44 included an updated version of the Heimdal Kerberos framework used by AuriStorFS when acquiring yfs-rxgk and rxkad authentication tokens. The updated Heimdal included a bug which disabled the use of DNS SRV records for KDC discovery and DNS TXT records for realm discovery. As a side effect token acquisition might fail with an unable to reach any KDC in realm error. This is fixed in v2021.05-44a.

v2021.05-44 (17 August 2024)

Cache Manager:
- Since v0.192 the cache manager has failed acquire the global lock when upgrading a shared-lock to a write-lock during the execution of a background cache chunk file truncation.
Authentication:
- Neither MIT nor Heimdal gssapi nor their gss mechanisms consistently initialize the output 'minorStatus' parameter. Various functions can return either success or failure majorStatus values with minorStatus unassigned. As a result, stack garbage will be used when generating error messages. From now on libyfs_acquire will always initialize the minorStatus output variable to zero before calling into the gssapi library.
Command Parser:
- No longer accept the token "-" as a switch which eventually fails with a CMD_UNKNOWNSWITCH error. Instead, process the token as a data value.
- Optimize the processing of the loop which processes "source" command input.
- If the source command input file is "-", read from stdin.

v2021.05-41 (26 June 2024)

Rx Networking (libyfs_rx):
- A race during event creation can lead to the freeing of the event while its still in use.
- RFC1122 says that Net and Host unreachable ICMP errors might be transient and should therefore not be treated as fatal. There is no such language for the equivalent ICMPV6 errors. However, in practice ICMP6_DST_UNREACH_NOROUTE, ICMP6_DST_UNREACH_BEYONDSCOPE, and ICMP6_DST_UNREACH_ADDR can be transient.
  Linux has considered these ICMPV6 destination unreachable errors as non-fatal going back at least as far as the initial git repository commit.
  AuriStor Rx has always treated these as fatal errors which results in immediate termination of in-flight calls when received. Even if the network route corrects itself before the call timeout period expires. This release mirrors the Linux behavior and makes these errors non-fatal.
Cache Manager:
- For the first time the cache manager can detect the deletion of a volume and handle the creation of a new volume with the same name but a different volume id.
- If the location service reports the deletion of a volume, invalidate all mount points to that volume.
- RXAFS_GetCapabilities RPC failures should not be treated as a fatal error preventing failover to another replica site.
Authentication ("libyfs_acquire") used by aklog, vos, pts, bos, afsio:
- rxkad_k5 token acquisition krb5 ccache management. This release altered the krb5 credential cache management strategy once again to work around different bugs in MIT krb5 and Heimdal.
- New ACQUIRE_ERR_CRED_EXPIRED error code introduced to represent the case when a request for a service credential returns one that is already expired.
Command parser (libyfs_cmd):
- When parsing configuration files there is a depth limit of ten active inclusions. This limit was improperly enforced as a limit of ten included files instead of a depth of ten included files. As of this release it is now possible to populate an includedir directory with any number of .conf files.

v2021.05-40

Not released.

v2021.05-39 (20 May 2024)

Parallel Random Number Generation:
- AuriStorFS processes rely upon the krb5_generate_random() and RAND_bytes() functions to obtain random bytes for cryptographic operations and random counters. krb5_generate_random() internally acquires a mutex to protect internal state information. This mutex has become a significant barrier to the encryption and checksumming of Rx packets with both yfs-rxgk and rxkad.
  
  This release replaces general use of krb5_generate_random() and RAND_bytes() with a per-thread ChaCha20 CS-PRNG. This avoids the acquisition of a global mutex and permits increased parallelism on multi-core systems.
Rx Networking (libyfs_rx):
- The Rx network stack schedules a garbage collection operation to execute once per minute. This operation enforces call timeouts, destroys idle connections and destroys idle peers. The operation has historically been performed by the Rx event thread which is already responsible for performing actions in response to call RTOs, sending NAT Ping and keep-alive packets, and retrying connection challenge and reachability checks.
  
  The time complexity of the garbage collection operation is determined by the number of calls, connections, and peers. The busier the Rx endpoint the more work must be performed during each garbage collection run and the longer it takes to complete. While garbage collection is active other events cannot be processed which can interfere with the proper flow control of active calls.
  
  As with all Rx events, the garbage collection event is scheduled to execute at an absolute clock time. If the system clock drifts (or is administratively set) backwards garbage collection will not be performed until the clock catches up with the scheduled time.
  
  Another responsibility of the garbage collection procedure is to terminate calls if the system clock drifted backwards by five minutes or longer. However, when the clocked drifts backwards garbage collection is not performed until the clock has advanced beyond the point where calls require termination. As a result, calls are not terminated due to backwards clock drift and they can stall.
  
  This release re-implements the garbage collection procedure using a dedicated thread and relative waits. This change ensures that the garbage collection procedure will not prevent the execution of call related events and permits calls to be terminated when large backward clock drifts are detected.
Disk Cache Management:
- Since IBM AFS 3.5, the cache has been considered "too full" even if there exist cache files have been discarded but not yet truncated. When the cache is "too full" most operations that write to the cache will block until truncation of discarded cache files has been performed which results in unnecessary delays. This release fixes the cache such that that discarded but not yet truncated cache files do not block write operations.
- This release permits the cache truncation daemon thread to exit sooner if the cache manager is shutting down.
Improved failover when the RXGK service (co-located with each vlserver) fails to issue tokens. The failures might be the result of misconfiguration, an inability to read keys or loss of Ubik quorum.

v2021.05-38 (29 February 2024)

As with other AuriStorFS releases since the beginning of 2024, this release includes additional improvements to the Rx RPC implementation which are related to the possibility of silent data corruption when Rx jumbograms are in use. Prior releases disabled the negotiation of Rx jumbograms such that the v2021.05-37 Rx peer will refuse to send Rx jumbograms and will request that the remote peer does not send them. However, a bad actor could choose to send Rx jumbograms even though they were asked not to. v2021.05-38 introduces additional protections to ensure that a corrupt Rx jumbogram is dropped instead of being accepted.

The v2021.05-38 Rx RPC implementation also includes two optimizations. First, when Rx initiators complete a call they will no longer send an extra ACK packet to the Rx acceptor of the completed call. The sending of this unnecessary ACK creates additional work for the server which can result in increased latency for other calls being processed by the server.

Second, all AuriStor Rx services require a reach check for incoming calls from Rx peers to help protect against Distributed Reflection Denial of Service (DRDoS) attacks and execution of RPCs when the response cannot be delivered to the caller. A new reach check is required for each new call that arrives more than 60 seconds after the prior reach check completed. v2021.05-38 Rx considers the successful acknowledgment of a response DATA packet as a reach check validation. With this change reach checks will not be periodically required for a peer that completes at least one call per 60 seconds. A 1 RTT delay is therefore avoided each time a reach check can be avoided. In addition, reach checks require the service to process an additional ACK packet. Eliminating a large number of reach checks can improve overall service performance.

The final Rx RPC change in this release is specific to kernel implementations. Prior releases restricted the frequency of executing time scheduled Rx events to a granularity no smaller than 500ms. As a result an RTO timer event for a lost packet could not be shorter than 500ms even if the measured RTT for the connection is significantly smaller. The minimum RTO for a connection in AuriStor Rx is 200ms. The inability to schedule shorter timeouts impacts recovery from packet loss.

v2021.05-37 (5 February 2024)

Rx improvements:
- The v2021.05-36 release permanently disabled all use of Rx jumbograms due to a risk of silent data corruption. However, when advertising the number of acceptable datagrams in the ACK trailer a missing htonl() set the value to 16777216 instead of 1 on little-endian systems.
- When sending a PING ACK as a reachability test, ensure that the previousPacket field is properly assigned to the largest accepted DATA packet sequence number instead of zero.
- Replace the initialization state flag with two flags. One that indicates that Rx initialization began and the other that it succeeded. The first prevents multiple attempts at initialization after failure. The second prevents shutdown from accessing uninitialized structures if initialization failed.
Cache Manager Improvements:
- No longer refuse to start if both the 'cachedir' and 'memcache' options are present in the configuration file.
- New variable to store the maximum number of cache blocks used. which is accessible via /proc/fs/auristorfs/cache/blocks_used_max.

v2021.05-36 (10 January 2024)

Rx improvements:
- Permanently disable all use of Rx jumbograms due to a risk of silent data corruption.
  
  Ever since OpenAFS 1.0, and possibly before, a race condition has existed when Rx transmits packets. As the rx_call.lock is dropped when starting packet transmission, there is no protection for data that is being copied into the kernel by sendmsg(). It is critical that this packet data is not modified by another thread. However, races exist between the application, listener, and event threads that can lead to retransmissions starting whilst an original transmission is still in progress. This can lead to the packet headers being overwritten, and either the original transmission, the retransmission or both sending corrupt data to the peer.
  
  This corruption can affect the packet serial number or packet flags. It is particularly harmful when the packet flags are corrupted, as this can lead to multiple Rx packets which were intended to be sent as Rx jumbograms being delivered and misinterpreted as a single large packet. The eventual result of this depends on the Rx security class in play, but it can cause decrypt integrity errors (rxgk:crypt and rxgk:auth) or corruption of the data stream (rxnull, rxgk:clear or rxkad:auth).
  
  All AuriStorFS servers, OpenAFS 1.6 or later servers, and the Windows cache manager have been shipped with Rx jumbograms disabled by default. The UNIX cache managers however are shipped with jumbograms enabled. There are many AFS cells around the world that continue to deploy OpenAFS 1.4 or earlier fileservers which continue to negotiate the use of Rx jumbograms.
  
  It is worth noting that all AuriStorFS v0.198 and later fileservers and cache managers implement explicit checks that will recognize the corrupted application data stream and prevent corrupted file content from being stored either into an AFS vnode's backing store in the volume's object store or the cache manager's AFS cache. OpenAFS cache managers and fileservers do not have these protections.
  
  With Rx jumbograms disabled the maximum number of Rx packets in a datagram is reduced from 6 to 1; the maximum number of send and receive datagram fragments is reduced from 4 to 1; and the maximum advertised MTU is restricted to 1444 - the maximum rx packet size prior to the introduction of jumbograms in IBM AFS 3.5.
- If the rx call flow state transitions from either the RECOVERY or RESCUE states to the LOSS state as a result of an RTO resend event while writing packets to the network, cease transmission of any new DATA packets if there are packets in the resend queue.
- When the call flow state is LOSS and all packets in the resend queue have been retransmitted and yet the recovery point has not been reached, then permit new DATA packets to be sent in order to maintain a full congestion window.
- Add a safety check to prevent the estimated RTT from underflowing when the actual roundtrip time is smaller than 125us.
- Fix the computation of the padding required for rxgk encrypted packets. This bug results in packets sending 8 bytes fewer per packets than the network permits. This bug accidentally prevented the construction of Rx jumbograms when a call is protected by rxgk:crypt.
- Replace the random number generator with a more security source of random bytes.

v2021.05-33 (27 November 2023)

Rx improvements:
- Not all calls transfer enough data to be able to measure a smoothed round-trip time (SRTT). Calls which are unable to compute a SRTT should not be used to update the peer host RTO value which is used to initialize the RTO for subsequent calls.
  
  Without this change, a single DATA packet call will cause the peer host RTO to be reduced to 0ms. Subsequent calls will start with a RTO value of MAX(0, rxi_minPeerTimeout) where rxi_minPeerTimeout defaults to 200ms. If the actual measured RTO is greater than 200ms, then initial RTO will be too small resulting in premature triggering of the RTO timer and the call flow state entering the loss phase which can significantly hurt performance.
- Initialize the peer host RTO to rxi_minPeerTimeout (which defaults to 200ms) instead of one second. Although RFC6298 recommends the use of one second when no SRTT is available, Rx has long used the rxi_minPeerTimeout value for other purposes which are supposed to be consistent with initial RTO value. It should be noted that Linux TCP uses 200ms instead of one second for this purpose.
- If associating a security class with an Rx connection fails immediately place the Rx connection into an error state. A failure might occur if the security class is unable to access valid key material.
- If an incoming Rx call requires authentication and the security class is unable to successfully generate a challenge, put the incoming Rx connection into an error state and issue an abort to the caller.
- If an incoming Rx call requires authentication and the security class is able to generate a challenge but the challenge cannot be returned to Rx, then treat this as a transient error. Do not acknowledge the incoming DATA packet and do not place the Rx connection into an error state. An attempt to re-issue the challenge will be performed when the DATA packet is retransmitted.
- If an Rx call is terminated due to the expiration of the configured connection dead time, idle dead time, hard dead time, or as a result of clock drift, then send an ABORT to the peer notifying them that the call has been terminated. This is particularly important for terminated outgoing calls. If the peer does not know to terminate the call, then the call channel might be in use when the next outgoing call is issued using the same call channel. If the next incoming call is received by an in-use call channel, the receiver must drop the received DATA packet and return a BUSY packet. The call initiator will need to wait for a retransmission timeout to pass before retransmitting the DATA packet. Receipt of BUSY packets cannot be used to keep a call alive and therefore the requested call is at greater risk of timing out if the network path is congested.
aklog and krb5.log (via libyfs_acquire):
- If the linked Kerberos library implements krb5_cc_cache_match() and libacquire has been told to use an explicit principal name and credential cache, the Kerberos library might return KRB5_CC_NOTFOUND even though the requested credential cache is the correct one to use. This release will not call krb5_cc_cache_match() if the requested credential cache contains the requested principal.
Cell Service Database (cellservdb.conf):
- cellservdb.conf has been synchronized with the 31 Oct 2023 update to the grand.central.org CellServDB file.

v2021.05-32 (9 October 2023)

No significant changes for macOS compared to v2021.05-31

v2021.05-31 (25 September 2023)

New platform:
- macOS 14 Sonoma
macOS 14 Sonoma:

AuriStorFS v2021.05-29 and later installers for macOS 13 Ventura are compatible with macOS 14 Sonoma and do not need to be removed before upgrading to macOS 14 Sonoma. Installation of the macOS 14 Sonoma version of AuriStorFS is recommended.

Cache Manager:
- If an AuriStorFS cache manager is unable to use the yfs-rxgk security class when communicating with an AuriStorFS fileserver, it must assume it is IBM AFS 3.6 or OpenAFS and upgrade it to AuriStorFS if an upgrade probe returns a positive result. Once a fileserver's type is identified as AuriStorFS the type should never be reset; even if communication with the fileserver is lost or the fileserver restarts.
  
  If an AuriStorFS fileserver is replaced by an OpenAFS fileserver on the same endpoint, then the UUID of the OpenAFS must be different. As a result, the OpenAFS fileserver will be observed as distinct from the AuriStorFS fileserver that previously shared the endpoint.
  
  Prior to this release there were circumstances in which the cache manager discarded the fileserver type information and would fail to recognize the fileserver as an AuriStorFS fileserver when yfs-rxgk could not be used. This release prevents the cache manager from resetting the type information if the fileserver is marked down.
- If a fileserver's location service entry is updated with a new uniquifier value (aka version number), this indicates that one of the following might have changed:
  1. the fileserver's capabilities
  2. the fileserver's security policy
  3. the fileserver's knowledge of the cell-wide yfs-rxgk key
  4. the fileserver's endpoints
  Beginning with this release the cache manager will force the establishment of new Rx connections to the fileserver when the uniquifier changes. This ensures that the cache manager will attempt to fetch new per-fileserver yfs-rxgk tokens from the cell's RXGK service, enforce the latest security policy, and not end up in a situation where its existing tokens cannot be used to communicate with the fileserver.
aklog:
- Fix incorrect output when populating the server list for a service fails. The stashed extended error explaining the cause of the failure was not displayed.
- If a cell has neither _afs3-prserver._udp. DNS SRV records nor AFSDB records, the lookup of the cell's protection servers would fail if there is no local cell configuration details. The fallback to use _afs3-vlserver._udp. DNS SRV records did not work. This is corrected in this release.

v2021.05-30 (6 September 2023)

Do not mark a fileserver down in response to a KRB5 error code.
fs cleanacl must not store back to the file server a cleaned acl if it was inherited from a directory. Doing so will create a file acl.
Correct the generation of never expire rxkad_krb5 tokens from Kerberos v5 tickets which must have a start time of Unix epoch and an end time of 0xFFFFFFFF seconds. The incorrectly generated tokens were subject to the maximum lifetime of 30 days.
Correct the generation of the yfs-rxgk RESPONSE packet header which failed to specify the key version generation number used to encrypt the authenticator. If the actual key version is greater than zero, then the authenticator would fail to verify.
Enforce a maximum NAT ping period of 20s to ensure that NAT/PAT/firewall rules due not expire while Rx RPCs are in-flight.

v2021.05-29 (26 June 2023)

Execution of fs commands such as examine, whereis, listquota, fetchacl, cleanacl, storeacl, whoami, lsmount, bypassthreshold and getserverprefs could result in memory leaks by the AuriStorFS kernel extension.

v2021.05-27 (1 May 2023)

Fixes for bugs in vos introduced in v2021.05-26.

v2021.05-26 (17 April 2023)

Fixed a potential kernel memory leak when triggered by fs examine, fs listquota, or fs quota.
Increased logging of VBUSY, VOFFLINE, VSALVAGE, and RX_RESTARTING error responses. A log message is now generated whenever a task begins to wait as a result of one of these error responses from a fileserver. Previously, a message was only logged if the volume location information was expired or discarded.
Several changes to optimize internal volume lookups.
Faster failover to replica sites when a fileserver returns RX_RESTARTING, VNOVOL or VMOVED.
rxdebug regains the ability to report rx call flags and rx_connection flags.
The RXRPC library now terminates calls in the QUEUED state when an ABORT packet is received. This clears the call channel making it available to accept another call and reduces the work load on the worker thread pool.
Fileserver endpoint registration changes no longer result in local invalidation of callbacks from that server.
Receipt of an RXAFSCB_InitCallBackState3 RPC from a fileserver no longer resets the volume site status information for all volumes on all servers.

v2021.05-25 (28 December 2022)

The v2021.05-25 release includes further changes to RXRPC to improve reliability. The changes in this release prevent improper packet size growth. Packet size growth should never occur when a call is attempting to recover from packet loss; and is unsafe when the network path's maximum transmission unit is unknown. Packet size growth with be re-enabled in a future AuriStorFS release that includes Path MTU detection and the Extended SACK functionality.
Improved error text describing the source of invalid values in /etc/yfs/yfs-client.conf or included files and directories.

v2021.05-24 (25 October 2022)

New Platform: macOS 13 (Ventura)

v2021.05-22 (12 September 2022) and v2021.05-21 (6 September 2022)

RX RPC
- If receipt of a DATA packet causes an RX call to enter an error state, do not send the ACK of the DATA packet following the ABORT packet. Only send the ABORT packet.
- AuriStor RX has failed to count and report the number of RX BUSY packets that have been sent. Beginning with this change the sent RX BUSY packet count is once again included in the statistics retrieved via rxdebug server port -rxstats.
- Introduce minimum and maximum bounds checks on the ACK packet trailer fields. If the advertised values are out of bounds for the receiving RX stack, do not abort the call but adjust the values to be consistent with the local RX RPC implementation limits. These changes are necessary to handle broken RX RPC implementations or prevent manipulation by attackers.

New to v2021.05-20 (15 August 2022) and v2021.05-19 (13 August 2022)

RX RPC
- Include the DATA packet serial number in the transmitted reachability check PING ACK. This permits the reachability test ACK to be used for RTT measurement.
- Do not terminate a call due to an idle dead timeout if there is data pending in the receive queue when the timeout period expires. Instead deliver the received data to the application. This change prevents idle dead timeouts on slow lossy network paths.
- Fix assignment of RX DATA, CHALLENGE, and RESPONSE packet serial numbers in macOS (KERNEL). Due to a mistake in the implementation of atomic_add_and_read the wrong serial numbers were assigned to outgoing packets.

New to v2021.05-18 (12 July 2022)

Cache Manager
- Prevent a kernel memory leak of less than 64 bytes for each bulkstat RPC issued to a fileserver. Bulkstat RPCs can be frequently issued and over time this small leak can consume a large amount of kernel memory. Leak introduced in AuriStorFS v0.196.
- The Perl::AFS module directly executes pioctls via the OpenAFS compatibility pioctl interface instead of the AuriStorFS pioctl interface. When Perl::AFS is used to store an access control list (ACL), the deprecated RXAFS_StoreACL RPC would be used in place of the newer RXAFS_StoreACL2 or RXYFS_StoreOpaqueACL2 RPCs. This release alters the behavior of the cache manager to use the newer RPCs if available on the fileserver and fallback to the deprecated RPC. The use of the deprecated RPC was restricted to use of the OpenAFS pioctl interface.
RX RPC
- Handle a race during RX connection pool probes that could have resulted in the wrong RX Service ID being returned for a contacted service. Failure to identify that correct service id can result in a degradation of service.
- The Path MTU detection logic sends padded PING ACK packets and requests a PING_RESPONSE ACK be sent if received. This permits the sender of the PING to probe the maximum transmission unit of the path. Under some circumstances attempts were made to send negative padding which resulted in a failure when sending the PING ACK. As a result, the Path MTU could not be measured. This release prevents the use of negative padding.
Preparation for supporting macOS 13 Ventura when it is released in Fall 2022.
Some shells append a slash to an expanded directory name in response to tab completion. These trailing slashes interfered with "fs lsmount", "fs flushmount" and "fs removeacl" processing. This release includes a change to prevent these commands from breaking when presented a trailing slash.

New to v2021.05-17 (16 May 2022)

Cell Service Database Updates
- Update cern.ch, ics.muni.cz, ifh.de, cs.cmu.edu, qatar.cmu.edu, it.kth.se
- Remove uni-hohenheim.de, rz-uni-jena.de, mathematik.uni-stuttgart.de, stud.mathematik.uni-stuttgart.de, wam.umd.edu
- Add ee.cooper.edu
- Restore ams.cern.ch, md.kth.se, italia
Fix parsing of [afsd] rxwindow configuration which can be used to specified a non-default send/receive RX window size. The current default is 128 packets.
RX Updates
- Add nPacketsReflected

Logging improvements

Cache directory validation errors log messages now include the cache directory path.

Log the active configuration path if "debug" logging is enabled.

More details of rxgk token extraction failures.

New to v2021.05-16 (24 March 2022)

RX - Previous releases re-armed the Retransmission Timeout (RTO) each time a new unacknowledged packet is acknowledged instead of when a new leading edge packet is acknowledged. If leading edge data packet and its retransmission are lost, the call can remain in the "recovery" state where it continues to send new data packets until one of the following is true:
. the maximum window size is reached
. the number of lost and resent packets equals 'cwind'
at which point there is nothing left to transmit. The leading edge data packet can only be retransmitted when entering the "loss" state but since the RTO is reset with each acknowledged packet the call stalls for one RTO period after the last transmitted data packet is acknowledged.

This poor behavior is less noticiable with small window sizes and short lived calls. However, as window sizes and round-trip times increase the impact of a twice lost packet becomes significant.

RX - Never set the high-order bit of the Connection Epoch field. RX peers starting with IBM AFS 3.1b through AuriStor RX v0.191 ignore the source endpoint when matching incoming packets to RX connections if the high-order epoch bit is set. Ignoring the source endpoint is problematic because it can result in a call entering a zombie state whereby all PING ACK packets are immediately responded to the source endpoint of the PING ACK but any delayed ACK or DATA packets are sent to the endpoint bound to the RX connection. An RX client that moves from one network to another or which has a NAT|PAT device between it and the service can find themselves stuck.

Starting with AuriStor RX v0.192 the high-order bit is ignored by AuriStor RX peer when receiving packets. This change to always clear the bit prevents IBM AFS and OpenAFS peers from ignoring the source endpoint.

RX - The initial packetSize calculation for a call is altered to require that all constructed packets before the receipt of the first ACK packet are eligible for use in jumbograms if and only if the local RX stack has jumbograms enabled and the maximum MTU is large enough. By default jumbograms are disabled for all AuriStorFS services. This change will have a beneficial impact if jumbograms are enabled via configuration; or when testing RX performance with "rxperf".

New fs whereis -noresolve option displays the fileservers by network endpoint instead of DNS PTR record hostname.

New to v2021.05-15 (24 January 2022)

kernel - fixed YFS_RXGK service rx connection pool leak

New to v2021.05-14 (20 January 2022)

fs mkmount permit mount point target strings longer than 63 characters.
afsd enhance logging of yfs-rxgk token renewal errors.

afsd gains a "principal = configuration option for use with keytab acquisition of yfs-rxgk tokens for the cache manager identity.

kernel - Avoid unnecessary rx connection replacement by racing threads after token replacement or expiration.

kernel - Fix a regression introduced in v2021.05 where an anonymous combined identity yfs-rxgk token would be replaced after three minutes resulting in the connection switching from yfs-rxgk to rxnull.

kernel - Fix a regression introduced in v0.208 which prevented the invalidation of cached access rights in response to a fileserver callback rpc. The cache would be updated after the first FetchStatus rpc after invalidation.

kernel - Reset combined identity yfs-rxgk tokens when the system token is replaced.

kernel - The replacement of rx connection bundles in the cache manager to permit more than four simultaneous rx calls per uid/pag with trunked rx connections introduced the following regressions in v2021.05.

a memory leak of discarded rx connection objects

failure of NAT ping probes after replacement of an connection

inappropriate use of rx connections after a service upgrade failure

All of these regressions are fixed in patch 14.

New to v2021.05-12 (7 October 2021)

fs ignorelist -type afsmountdir in prior releases could prevent access to /afs.

Location server rpc timeout restored to two minutes instead of twenty minutes.

Location server reachability probe timeout restored to six seconds instead of fifty seconds.

Cell location server upcall results are now cached for fifteen seconds.

Multiple kernel threads waiting for updated cell location server reachability probes now share the results of a single probe.

RX RPC implementation lock hierarchy modified to prevent a lock inversion.

RX RPC client connection reference count leak fixed.

RX RPC deadlock during failed connection service upgrade attempt fixed..

New to v2021.05-9 (25 October 2021)

First public release for macOS 12 Monterey build using XCode 13. When upgrading macOS to Monterey from earlier macOS releases, please upgrade AuriStorFS to v2021.05-9 on the starting macOS release, upgrade to Monterey and then install the Monterey specific v2021.05-9 release.

Improved logging of "afsd" shutdown when "debug" mode is enabled.

Minor RX network stack improvements

New to v2021.05-3 (10 June 2021)

Fix for [cells] cellname = {...} without server list.

New to v2021.05 (31 May 2021)

Multi-homed location servers are finally managed as a single server instead of treating each endpoint as a separate server. The new functionality is a part of the wholesale replacement of the former cell management infrastructure. Location server communication is now entirely managed as a cluster of multi-homed servers for each cell. The new infrastructure does not rely upon the global lock for thread safety.

This release introduces a new infrastructure for managing user/pag entities and tracking their per cell tokens and related connection pools.

Expired tokens are no longer immediately deleted so that its possible for them to be listed by "tokens" for up to two hours.

Prevent a lock inversion introduced in v0.208 that can result in a deadlock involving the GLOCK and the rx call.lock. The deadlock can occur if a cell's list of location servers expires and during the rebuild an rx abort is issued.

Add support for rxkad "auth" mode rx connections in addition to "clear" and "crypt". "auth" mode provides integrity protection without privacy.

Add support for yfs-rxgk "clear" and "auth" rx connection modes.

Do not leak a directory buffer page reference when populating a directory page fails.

Re-initialize state when populating a disk cache entry using the fast path fails and a retry is performed using the slow path. If the data version changes between the attempts it is possible for truncated disk cache data to be treated as valid.

Log warnings if a directory lookup operation fails with an EIO error. An EIO error indicates that an invalid directory header, page header, or directory entry was found.

Do not overwrite RX errors with local errors during Direct-I/O and StoreMini operations. Doing so can result in loss of VBUSY, VOFFLINE, UAENOSPC, and similar errors.

Correct a direct i/o code path which could overwrite a fileserver returned error code with a local RXGEN_CC_UNMARSHAL error.

Correct the StoreMini code path which could overwrite a fileserver returned error code with a local RXGEN_CC_UNMARSHAL error.

Ensure the rx call object is not locked when writing to the network socket.

Removed all knowledge of the KERNEL global lock from RX. Acquiring the GLOCK from RX is never safe if any other lock is held. Doing so is a lock order violation that can result in deadlocks.

Fixed a race in the opr_reservation system that could produce a cache entry reference undercount.

If a directory hash chain contains a circular link, a buffer page reference could be leaked for each traversal.

Each AFS3 directory header and page header contains a magic tag value that can be used in a consistency check but was not previously checked before use of each header. If the header memory is zero filled during a lookup, the search would fail producing an ENOENT error. Starting with this release the magic tag values are validated on each use. An EIO error is returned if there is a tag mismatch.

"fs setcrypt -crypt auth" is now a permitted value. The "auth" mode provides integrity protection but no privacy protection.

Add new "aklog -levels " option which permits requesting "clear" and "auth" modes for use with yfs-rxgk.

Update MKShim to Apple OpenSource MITKerberosShim-79.

Report KLL errors via a notification instead of throwing an exception which (if not caught) will result in process termination.

If an exception occurs while executing "unlog" catch it and ignore it. Otherwise, the process will terminate.

New to v2021.04 (22 April 2021)

Primarily bug fixes for issues that have been present for years.

A possibility of an infinite kernel loop if a rare file write / truncate pattern occurs.

A bug in silly rename handling that can prevent cache manager initiated garbage collection of vnodes.

New to v0.209 (13 March 2021)

fs setserverprefs and fs getserverprefs updated to support IPv6 and CIDR specifications.

Improved error handling during fetch data and store data operations.

Prevents a race between two vfs operations on the same directory which can result in caching of out of date directory contents.

Use cached mount point target information instead of evaluating the mount point's target upon each access.

Avoid rare data cache thrashing condition.

Prevent infinite loop if a disk cache error occurs after the first page in a chunk is written.

Network errors are supposed to be returned to userspace as ETIMEDOUT. Previously some were returned as EIO.

When authentication tokens expire, reissue the fileserver request anonymously. If the anonymous user does not have permission either EACCES or EPERM will be returned as the error to userspace. Previously the vfs request would fail with an RXKADEXPIRED or RXGKEXPIRED error.

If growth of an existing connection vector fails, wait on a call slot in a previously created connection instead of failing the vfs request.

Volume and fileserver location query infrastructure has been replaced with a new modern implementation.

Replace the cache manager's token management infrastructure with a new modern implementation.

New to v0.206 (12 January 2021) - Bug fixes

Prevents a possible panic during unmount of /afs.

Improved failover and retry logic for offline volumes.

New to v0.205 (24 December 2020) - Bug fixes

Volume name-to-id cache improvements

Fix expiration of name-to-id cache entries

Control volume name-to-id via sysctl

Query volume name-to-id statistics via sysctl

Improve error handling for offline volumes

Fix installer to prevent unnecessary installation of Rosetta 2 on Apple Silicon

New to v0.204 (25 November 2020) - Bug fix for macOS Big Sur

v0.204 prevents a kernel panic on Big Sur when AuriStorFS is stopped and restarted without an operating system reboot.

introduces a volume name-to-id cache independent of the volume location cache.

New to v0.203 (13 November 2020) - Bug fix for macOS

v0.203 prevents a potential kernel panic due to network error.

New to v0.201 (12 November 2020) - Universal Big Sur (11.0) release for Apple Silicon and Intel

v0.201 introduces a new cache manager architecture on all macOS versions except for High Sierra (10.12). The new architecture includes a redesign of:

kernel extension load

kernel extension unload (not available on Big Sur)

/afs mount

/afs unmount

userspace networking

The conversion to userspace networking will have two user visible impacts for end users:
The Apple Firewall as configured by System Preferences -> Security & Privacy -> Firewall is now enforced. The "Automatically allow downloaded signed software to receive incoming connections" includes AuriStorFS.

Observed network throughput is likely to vary compared to previous releases.

On Catalina the "Legacy Kernel Extension" warnings that were displayed after boot with previous releases of AuriStorFS are no longer presented with v0.201.

AuriStorFS /afs access is expected to continue to function when upgrading from Mojave or Catalina to Big Sur. However, as AuriStorFS is built specifically for each macOS release, it is recommended that end users install a Big Sur specific AuriStorFS package.
AuriStorFS on Apple Silicon supports hardware accelerated aes256-cts-hmac-sha1-96 and aes128-cts-hmac-sha1-96 using AuriStor's proprietary implementation.

New to v0.200 (4 November 2020) - Final release for macOS El Capitan (10.11)

The network path between a client and a server often traverses one or more network segments separated by NAT/PAT devices. If a NAT/PAT times out a RPCs endpoint translation mid-call, this can result in an extended delay before failure and the server being marked down, or worse, a call that never terminates and a client that appears to hang until the fileserver is restarted.

This release includes significant changes to the RX stack and the UNIX cache manager to detect such conditions, fail the calls quickly and detect when it is safe to retry the RPC.

NAT/PAT devices that drop endpoint mappings while in use are anti-social and can result in unwanted delays and even data loss. Whenever possible they should be avoided when they can be. That said, the changes in this release are a huge step to making the loss of endpoint mappings tolerable.

Fix segmentation fault of Backgrounder when krb5_get_credentials() fails due to lack of network connectivity.

Fix the "afsd" rxbind option which was ignored if the default port, 7001, is in use by another process on the system.

If a direct i/o StoreData or FetchData RPC failed such that it must be retried, the retried RPC would fail due to an attempt to Fetch or Store the wrong amount of data. This is fixed.

Servers are no longer marked down if RPCs fail with RX_CALL_PEER_RESET, RX_CALL_EXCEEDS_WINDOW, or RX_PROTOCOL_ERROR. RPCs that are safe to retry are retried.

Fixed a race between a call entering error state and call completion that can result in a the call remaining in the DALLY state and the connection channel remaining in use. If this occurs during process or system shutdown it can result in a deadlock.

During shutdown cancel any pending delayed aborts to prevent a potential deadlock. If a deadlock occurs when unloading a kernel module a reboot will be required.

Updated cellservdb.conf

New to v0.197.1 (31 August 2020) and v0.198 (10 October 2020)

Prevent Dead vnode has core/unlinkedel/flock panic introduced in v0.197.

New to v0.197 (26 August 2020)

A new callback management framework for UNIX cache managers reduces the expense of processing volume callback RPCs from O(number of vcache objects) to O(1). A significant amount of lock contention has been avoided. The new design reduces the risk of the single callback service worker thread blocking. Delays in processing callbacks on a client can adversely impact fileserver performance and other clients in the cell.

Bulk fetch status RPCs are available on macOS for the first time. Bulk fetch status permits optimistic caching of vnode status information without additional round-trips. Individual fetch status RPCs are no longer issued if a bulk status fails to obtain the required status information.

Hardware accelerated crypto is now available for macOS cache managers. AuriStor's proprietary aes256-cts-hmac-sha1-96 and aes256-cts-hmac-sha512-384 implementations leverage Intel processor extensions: AESNI AVX2 AVX SSE41 SSSE3 to achieve the fastest encrypt, decrypt, sign and verify times for RX packets.

This release optimizes the removal of "._" files that are used to store extended attributes by avoiding unnecessary status fetches when the directory entry is going to be removed.

When removing the final directory entry for an in-use vnode, the directory entry must be silly renamed on the fileserver to prevent removal of the backing vnode. The prior implementation risked blindly renaming over an existing silly rename directory entry.

Behavior change! When the vfs performs a lookup on ".", immediately return the current vnode.

if the object is a mount point, do not perform fakestat and attempt to resolve the target volume root vnode.

do not perform any additional access checks on the vnode. If the caller already knows the vnode the access checks were performed earlier. If the access rights have changed, they will be enforced when the vnode is used just as they would have if the lookup of "." was performed within the vfs.

do not perform a fetch status or fetch data rpcs. Again, the same as if the lookup of "." was performed within the vfs.

Volumes mounted at more than one location in the /afs namespace are problematic on more than one operating system that do not expect directories to have more than one parent. It is particularly problematic if a volume is mounted within itself. Starting with this release any attempt to traverse a mountpoint to the volume containing the mountpoint will fail with ENODEV.

When evaluating volume root vnodes, ensure that the vnode's parent is set to the parent directory of the traversed mountpoint and not the mountpoint. Vnodes without a parent can cause spurious ENOENT errors on Mojave and later.

v0.196 was not publicly released.

New to v0.195 (14 May 2020)

This is a CRITICAL update for AuriStorFS macOS clients.

In Sep 2019 AuriStorFS v0.189 was released which provided faster and less CPU intensive writing of (>64GB) large files to /afs. These improvements introduced a hash collision bug in the store data path of the UNIX cache manager which can result in file corruption. If a hash collision occurs between two or more files that are actively being written to via cached I/O (not direct I/O), dirty data can be discarded from the auristorfs cache before it is written to the fileserver creating a file with a range of zeros (a hole) on the fileserver. This hole might not be visible to the application that wrote the data because the lost data was cached by the operating system. This bug has been fixed in v0.195 and it is for this reason that v0.195 has been designated a CRITICAL release for UNIX/Linux clients.

While debugging a Linux SIGBUS issue, it was observed that receipt of an ICMP network error in response to a transmitted packet could result in termination of an unrelated rx call and could mark a server down. If the terminated call is a StoreData RPC, permanent data loss will occur. All Linux clients derived from the IBM AFS code base experience this bug. The v0.195 release prevents this behavior.

This release includes changes that impact all supported UNIX/Linux cache managers. On macOS there is reduced lock contention between kernel threads when the vcache limit has been reached.

The directory name lookup cache (DNLC) implementation was replaced. The new implementation avoids the use of vcache pointers which did not have associated reference counts, and eliminates the invalidation overhead during callback processing. The DNLC now supports arbitrary directory name lengths; the prior implementation only cached entries with names not exceeding 31 characters.

Prevent matching arbitrary cell name prefixes as aliases. For example "/afs/y" should not be an alias for "your-file-system.com". Some shells, for example "zsh", query the filesystem for names as users type. Delays between typed characters result in filesystem lookups. When this occurs in the /afs dynroot directory, this could result in cellname prefix string matches and the dynamic creation of directory entries for those prefixes.

New to v0.194 (2 April 2020)

This is a CRITICAL release for all macOS users. All prior macOS clients whether AuriStorFS or OpenAFS included a bug that could result in data corruption either when reading or writing.

This release also fixes these other issues:

sign and notarize installer plugin "afscell" bundle. The lack of digital signature prevented the installer from prompting for a cellname on some macOS versions.

prevent potential for corruption when caching locally modified directories.

v0.193 was withdrawn due to a newly introduced bug that could result in data corruption.

New to v0.192 (30 January 2020)

The changes improve stability, efficiency, and scalability. Post-0.189 changes exposed race conditions and reference count errors which can lead to a system panic or deadlock. In addition to addressing these deficiencies this release removes bottlenecks that restricted the number of simultaneous vfs operations that could be processed by the AuriStorFS cache manager. The changes in this release have been successfully tested with greater than 400 simultaneous requests sustained for for several days.

New to v0.191 (16 December 2019)

Restore keyed cache manager capability broken in v0.189.

Add kernel module version string to AuriStorFS Preference Pane.

Other kernel module bug fixes.

New to v0.190 (14 November 2019)

Short-circuit busy volume retries after volume or volume location entry is removed.

New to v0.189 (28 October 2019)

Faster "git status" operation on repositories stored in /afs.

Faster and less CPU intensive writing of (>64GB) large files to /afs. Prior to this release writing files larger than 1TB might not complete. With this release store data throughput is consistent regardless of file size. (See "UNIX Cache Manager large file performance improvements" later in this file).

macOS Catalina (8 October 2019)

AuriStorFS v0.188 released for macOS Catalina (10.15)

New to v0.188 (23 June 2019)

Increased clock resolution for timed waits from 1s to 1ns

Added error handling for rx multi rpcs interrupted by signals

New to v0.186 (29 May 2019)

v0.184 moved the /etc/yfs/cmstate.dat file to /var/yfs. With this change afsd would fail to start if /etc/yfs/cmstate.dat exists but contains invalid state information. This is fixed.

v0.184 introduced a potential deadlock during directory processing. This is fixed.

Handle common error table errors obtained outside an afs_Analyze loop. Map VL errors to ENODEV and RX, RXKAD, RXGK errors to ETIMEDOUT

Log all server down and server up events. Transition events from server probes failed to log messages.

RX RPC networking:

If the RPC initiator successfully completes a call without consuming all of the response data fail the call by sending an RX_PROTOCOL_ERROR ABORT to the acceptor and returning a new error, RX_CALL_PREMATURE_END, to the initiator.
Prior to this change failure to consume all of the response data would be silently ignored by the initiator and the acceptor might resend the unconsumed data until any idle timeout expired. The default idle timeout is 60 seconds.

Avoid transmitting ABORT, CHALLENGE, and RESPONSE packets with an uninitialized sequence number. The sequence number is ignored for these packets but set it to zero.

New to v0.184 (26 March 2019)

The initial congestion window has been reduced from 10 Rx packets to 4. Packet reordering and loss has been observed when sending 10 Rx packets via sendmmsg() in a single burst. The lack of udp packet pacing can also increase the likelihood of transmission stalls due to ack clock variation.

The UNIX Cache Manager underwent major revisions to improve the end user experience by revealing more error codes, improving directory cache efficiency, and overall resiliency. The cache manager implementation was redesigned to be more compatible with operating systems such as Linux and macOS that support restartable system calls. With these changes errors such as "Operation not permitted", "No space left on device", "Quota exceeded", and "Interrupted system call" can be reliably reported to applications. Previously such errors might have been converted to "I/O error".

New to v0.180 (9 November 2018)

RX reliability and performance improvements for high latency and/or lossy network paths such as public wide area networks.

A fix for a macOS firewall triggered kernel panic introduced in v0.177.

New to v0.177 (17 October 2018)

A fix to AuriStor's RX implementation bug introduced in v0.176 that interferes with communication with OpenAFS and IBM Location and File Services.

New to v0.176 (3 October 2018)

AuriStor's RX implementation has undergone a major upgrade of its flow control model. Prior implementations were based on TCP Reno Congestion Control as documented in RFC5681; and SACK behavior that was loosely modelled on RFC2018. The new RX state machine implements SACK based loss recovery as documented in RFC6675, with elements of New Reno from RFC5682 on top of TCP-style congestion control elements as documented in RFC5681. The new RX also implements RFC2861 style congestion window validation.

When sending data the RX peer implementing these changes will be more likely to sustain the maximum available throughput while at the same time improving fairness towards competing network data flows. The improved estimation of available pipe capacity permits an increase in the default maximum window size from 60 packets (84.6 KB) to 128 packets (180.5 KB). The larger window size increases the per call theoretical maximum throughput on a 1ms RTT link from 693 mbit/sec to 1478 mbit/sec and on a 30ms RTT link from 23.1 mbit/sec to 49.39 mbit/sec.

Improve shutdown performance by refusing to give up callbacks to known unreachable file servers and apply a shorter timeout period for the rest.

Permit RXAFSCB_WhoAreYou to be successfully executed after an IBM AFS or OpenAFS fileserver unintentionally requests an RX service upgrade from RXAFSCB to RXYFSCB.

RXAFS timestamps are conveyed in unsigned 32-bit integers with a valid range of 1 Jan 1970 (Unix Epoch) through 07 Feb 2106. UNIX kernel timestamps are stored in 32-bit signed integers with a valid range of 13 Dec 1901 through 19 Jan 2038. This discrepency causes RXAFS timestamps within the 2038-2106 range to display as pre-Epoch.

RX Connection lifecycle management was susceptible to a number of race conditions that could result in assertion failures, the lack of a NAT ping connection to each file server, and the potential reuse of RX connections that should have been discarded.

This release includes a redesigned lifecycle that is thread safe, avoids assertions, prevents NAT ping connection loss, and ensures that discarded connections are not reused.

The 0.174 release unintentionally altered the data structure returned to xstat_cm queries. This release restores the correct wire format.

Since v0.171, if a FetchData RPC fails with a VBUSY error and there is only one reachable fileserver hosting the volume, then the VFS request will immediately with an ETIMEDOUT error ("Connection timed out").

v0.176 corrects three bugs that contributed to this failure condition. One was introduced in v0.171, another in 0.162 and the final one dates to IBM AFS 3.5p1.

The intended behavior is that a cache manager, when all volume sites fail an RPC with a VBUSY error, will sleep for up to 15 seconds and then retry the RPC as if the VBUSY error had never been received. If the RPC continues to receive VBUSY errors from all sites after 100 cycles, the request will be failed with EWOULDBLOCK ("Operation would block") and not ETIMEDOUT.

Prefer VOLMISSING and VOLBUSY error states to network error states when generating error codes to return to the VFS layer. This will result in ENODEV ("No such device") errors when all volume sites return VNOVOL or VOFFLINE errors and EWOULDBLOCK ("Operation would block") errors when all volume sites return VBUSY errors. (v0.176)

New to v0.174 (24 September 2018)

macOS Mojave (10.14) support

New to v0.170 (27 April 2018)

Faster processing of cell configuration information by caching service name to port information.

RX call sequence number rollover to permit calls that require the transmission of more than 5.5TB of data.

Command parser Daylight Saving Time bug fix

Fix a bug that prevented immediate access to a mount point created with "fs mkmount" on the same machine.

Fix the setting of "[afsd] sysnames = " during cache manager startup.

New to v0.168 (6 March 2018)

Corrects "fs setacl -negative" processing [CVE-2018-7168]

Improved reliability for keyed cache managers. More persistent key acquisition renewals.

Major refresh to cellservdb.conf contents.

DNS SRV and DNS AFSDB records now take precedence when use_dns = yes

Kerberos realm hinting provided by
kerberos_realm = [REALM]

DNS host names are resolved instead of reliance on hard coded IP addresses
The cache manager now defaults to sparse dynamic root behavior. Only thiscell and those cells that are assigned aliases are included in /afs directory enumeration at startup. Other cells will be dynamically added upon first access.
Several other quality control improvements.

New to v0.167 (7 December 2017)

Addresses a critical remote denial of service vulnerability [CVE-2017-17432]
Alters the volume location information expiration policy to reduce the risk of single points of failures after volume release operations.
'fs setquota' when issued with quota values larger than 2TB will fail against OpenAFS and IBM AFS file servers
Memory management improvements for the memory caches.

New to v0.164 (11 November 2017)

Internal cache manager redesign. No new functionality.

New to v0.160 (21 September 2017)

Support for OSX High Sierra's new Apple File System (APFS). Customers must upgrade to v0.160 or later before upgrading to OSX High Sierra.
Reduced memory requirements for rx listener thread
Avoid triggering a system panic if an AFS local disk cache file is deleted or becomes inaccessible.
Fixes to "fs" command line output

New to v0.159 (7 August 2017)

Improved failover behavior during volume maintenance operations
Corrected a race that could lead the rx listener thread to enter an infinite loop and cease processing incoming packets.

New to v0.157 (12 July 2017)

Bundled with Heimdal 7.4 to address CVE-2017-11103 (Orpheus' Lyre puts Kerberos to sleep!)
"vos" support for volume quotas larger than 2TB.
"fs flushvolume" works
Fixed a bug that can result in a system panic during server capability testing

New to v0.150

AuriStorFS file server detection improvements

New to v0.149

rxkad encryption is enabled by default. Use "fs setcrypt off" to disable encryption when tokens are available.
Fix a bug in atomic operations on Sierra and El Capitan which could adversely impact Rx behavior.

New to v0.128

Extended attribute ._ files are automatically removed when the associated files are unlinked
Throughput improvements when sending data

New to v0.121

OSX Sierra support

New to v0.117

Cache file moved to a persistent location on local disk
AuriStor File System graphics
Improvements in Background token fetch functionality
Fixed a bug introduced in v0.44 that could result in an operating system crash when enumerating AFS directories containing Unicode file names (v0.106)
El Capitan security changes prevented Finder from deleting files and directories. As of v0.106, the AuriStor OSX client implements the required functionality to permit the DesktopHelperService to securely access the AFS cache as the user permitting Finder to delete files and directories.

Features:

Not vulnerable to OPENAFS-SA-2015-007.
Office 2011 can save to /afs.
Office 2016 can now save files to /afs.
OSX Finder and Preview can open executable documents without triggering a "Corrupted File" warning. .AI, .PDF, .TIFF, .JPG, .DOCX, .XLSX, .PPTX, and other structured documents that might contain scripts were impacted.
All file names are now stored to the file server using Unicode UTF-8 Normalization Form C which is compatible with Microsoft Windows.
All file names are converted to Unicode UTF-8 Normalization Form D for processing by OSX applications.

Known issues:

None

macOS Installer (10.14 Mojave)

Release Notes

Known Issues

If the Kerberos default realm is not configured, a delay of 6m 59s can occur before the AuriStorFS Backgrounder will acquire tokens and display its icon in the macOS menu. This is the result of macOS performing a Bonjour (MDNS) query in an attempt to discover the local realm.
"aklog" will fail to acquire tokens if the current working directory is located in an inaccessible AFS volume. When the "cwd" is inaccessible, the Heimdal framework is unable to initialize the Kerberos v5 context and reports an out of memory error.

New v2021.05-64 (22 Aug 2025)

The v2021.05-64 release fixes a race condition which can lead to a kernel panic when executing the "unlog" command.

v2021.05-63 (31 May 2025)

When AuriStorFS cache managers added support for per-file ACLs in v0.1, the masking of access rights based upon the inode's S_IRWXU mode bits were skipped when reading objects from AuriStorFS fileservers. The masking continued to be applied when reading objects from OpenAFS fileservers. This release applies S_IRWXU mode bits masking of granted access rights to both AuriStorFS and OpenAFS fileservers.
The existing fake stat mountpoint evaluation logic assumes that the fileserver returned a AFSFetchStatus.ParentVnode that refers to the parent directory from which the mountpoint was located. However, this is only guaranteed to be true if the mountpoint does not have a per-file ACL (which is permitted) and has never been cross-directory hard linked (which is also permitted).
This release improves the error handling logic to fail the fake stat evaluation if the AFSFetchStatus.ParentVnode does not match expectations.
This release fixes a metadata inconsistency when a mountpoint is created when disconnected mode is enabled. A mountpoint should be reported to the vfs as a directory and not a symlink.
The cache manager must track for each inode whether the object's status infomration was received from an AFS3 fileserver or an AuriStorFS fileserver. This release alters when that information is recorded to ensure that it is recorded each time the inode's status information is updated by a fileserver response.
This knowledge is required to properly determine how to interpret the status information.
Directory entries referring to mountpoints and volume roots must not be stored within the Directory Name Lookup Cache (DNLC). Instead, full lookups must be performed each time to ensure proper mountpoint resolution.
Prior to this release locally created mountpoints were stored into the DNLC by mistake.

v2021.05-62 (24 April 2025)

macOS Sequoia v15.4 vs Cisco AnyConnect VPN

Beginning with macOS 15.4, use of Cisco AnyConnect VPN breaks DNS SRV, AFSDB and TXT record lookups performed using the cross-platform DNS resolver API. As a result, clients which rely upon DNS-based zero-configuration resolution of Kerberos realms and/or AFS cells database services are unable to access those resources. This release switches the macOS builds to use the Apple DNS API.
The lookup failures impacted all command line tools, clients and services.
Note: Communication failures can occur if the VPN is configured by the organization to block the transmission of fragmented UDP packets. Set [afsd] rxmaxmtu = 1328 to ensure all RX packets are small enough to traverse the VPN link.
This release improves consistency of vnode metadata interpretation when the status information is received via an RPC which does not return a callback.

v2021.05-61 (31 March 2025)

Instead of allocating the same sized hash table for the vcache and dcache objects regardless of how many elements are expected to be stored, scale the hash table sizes. The vcache hash table will scale from 4096 buckets to 131072 buckets with a target of 4 entries per bucket. The dcache hash tables will scale from 4096 buckets to 65536 buckets with a target of 2 entries per bucket.
Historically, afs cache managers which create symlinks (including mountpoints) locally allocate a fake infinite lifetime callback to permit applications to use the symlink (or mountpoint) without requiring FetchStatus RPCs to be issued to the fileserver. This approach has drawbacks because it means that fileservers will not issue a callback break when the volume containing the symlink is taken offline, moved, deleted, etc. It also means that a callback will not be sent if another cache manager removes the symlink.
Beginning with this release the AuriStorFS cache manager will no longer allocate a fake infinite lifetime callback and will instead rely upon fileserver allocated callbacks.
"fs makemount" is supposed to create a mountpoint even if the target volume does not exist or is inaccessible. v2021.05 introduced support for creating mountpoints which kafs which broke the ability to create mountpoints to non-existent volumes.

v2021.05-60 (6 March 2025)

The cache manager retries fileserver RPCs up to 100 times when the fileserver replies with VBUSY, VOFFLINE or VSALVAGE. The cache manager fails the application request after the limit is reached. Prior to this release, the failure would result in the cache manager returning an ENODEV error to the application. But ENODEV indicates that the required volume does not exist and the fileserver has not indicated that the volume does not exist. The cache manager will now return EWOULDBLOCK to the application because the volume might be temporarily inaccessible.
Many OpenAFS and AuriStorFS fileservers can return VNOVOL as a side effect of a race between a volume going offline so that control can be given to the Volume Management Service (volserver) and an incoming RPC requesting that the volume be attached. A VNOVOL can also be returned because the volume was temporarily deleted from the vice partition prior to being replaced by a new clone, or prior to the fileserver marking the volume group as having been moved to another fileserver.
The fileserver replying to an RPC with VNOVOL cannot be trusted to mean that the volume has been permanently removed unless it is matched by the Location Service reporting the Volume Location (VL) entry has been deleted. From now on when a VNOVOL response is received and the VL entry exists, the cache manager will return EWOULDBLOCK instead of ENODEV to the application.
Prior to this release, any VL error code returned from the Location Service would be interpreted as meaning that the requested VL entry does not exist. Starting with this release only the VL_NOENT error will be interpreted as the VL entry does not exist.
Restrict the internal use of the ENOENT error code so the only use of ENOENT is to indicate that a directory entry does not exist. Previously, ENOENT could be returned if a disk cache entry could not be allocated or found. A disk cache related ENOENT returned when processing a directory might have been confused for the requested directory entry not existing.
Restrict the internal use of the ENODEV error code so that the only meaning of ENODEV is that the required volume does not exist. Previously, ENODEV could be returned due to transient errors when failing to contact the Location Service.

v2021.05-59 (13 February 2025)

The fix for garbage collection of idle unixuser objects introduced in v2021.05-56 was incomplete. As a result, Rx client connections might not be garbage collected.

v2021.05-57 (9 February 2025)

No new changes for macOS.

v2021.05-56 (7 February 2025)

Minor bug fixes to the Rx RPC implementation
The fix for garbage collection of idle unixuser objects introduced in v2021.05-55 was incomplete. As a result, Rx client connections might not be garbage collected.

v2021.05-55 (23 January 2025)

Minor bug fixes to the Rx RPC implementation
During garbage collection of idle unixuser objects, any Rx client connections associated with the unixuser (uid or pag) that were in use by another thread would not have their cache manager reference dropped. As a result the Rx client connections could never be garbage collected which is problematic on systems which churn through many unixusers. The behavior has been changed so that unixusers objects with in use Rx connections will not be freed. This permits future attempts to garbage collect unixusers to do so only after the Rx connections can also be freed.
Although this misbehavior has been present for decades, it is only since v2021.05-52 that the problem has been observed. It is possible that the changes to reduce the holding of the GLOCK has created a window of opportunity for connections to be simultaneously in use during unixuser garbage collection.

v2021.05-54 (19 January 2025)

The UNIX cache manager changes continue AuriStor's efforts to address a report that was received at the end of September. The end user site uses an OpenAFS cell to distribute software for execution by IBM Spectrum LSF (load sharing facility) managed compute nodes. Each of the approximately one dozen compute nodes is configured with approximately 100 processor cores and 1TB of ram. Each job scheduled by LSF runs in a newly allocated Process Authentication Group (PAG). Some jobs run for hours or days; others complete in a few seconds. LSF has been observed scheduling more than 3000 jobs in under a minute with peak counts of cache manager user objects at 1300. Everything ran well until it didn't. After some unexplained trigger event, all of the remaining free memory on the system would be consumed in about an hour causing the system to panic.

Over the last five months AuriStor has diligently worked to develop theories, optimize the AuriStorFS UNIX cache manager to address each theory; then move on to subsequent ideas once prior theories were disproved. Over this time we have replaced the memory management lifecycle for access rights entries and rx events. We have made it possible for Linux permission checks to be processed without acquiring the GLOCK in the majority of access patterns thereby increasing parallelism. The lifecycle management of PAG allocation and user entity tracking has been rewritten to avoid the GLOCK. The Linux inode lifecycle management has been rewritten which some sites attribute to elimination of swapping on their systems. All of these changes increased throughput and parallelism within and without the cache manager. However, none of them prevented the memory consumption once the unknown trigger event occurred.

It is with great pleasure that we can report that the trigger event and the source of the memory consumption have been identified. The trigger is a damaged Rx packet header which causes the AuriStor Rx stack to place an Rx connection into an RX_PROTOCOL_ERROR error state when there is an in-flight call to an OpenAFS fileserver. The memory consumption is due to the infinite loop that occurs when an Rx connection is in an error state and the error code is not one of the error codes which forces replacement of the Rx connection. Within each pass of the loop a structure is allocated to track the history of per-server errors so they can be logged when hard mount mode is enabled for the cell. AuriStor Rx permits up to 12 calls on each Rx connection. As a result there can be up to a dozen processes each spinning and allocating memory when the Rx connection is placed into the error state.

Days before Christmas 2023 AuriStor identified a race condition which can result in damaged Rx DATA packets being written to the network. Initial transmission of Rx DATA packets can be sent either from an application thread or an Rx listener. Packet retransmission occurs either in an Rx listener thread or in the Rx event thread. The race permits a second thread to modify the Rx packet headers while the sendmsg() syscall from the original transmission is executing. This race is possible because the original thread might exhaust its timeslice during the sendmsg() call and it might be many milliseconds before it is rescheduled on a busy system. In the meantime, packet retransmissions might be initiated either by an Rx listener thread processing an ACK packet or the Rx event thread processing a timeout event. If the initial packet transmission completes while the Rx headers are being rewritten the resulting packet sent on the wire might be garbage. AuriStor observed that if the modified packets were jumbograms that silent data corruption could occur so we disabled all use of jumbograms in v2021.05-35 We also notified both IBM and OpenAFS of the problem.

v2021.05-53 modified the RPC result analyze logic to force the replacement of Rx connections in an error state except when the error signals a token expiration. When a token is expired there is no benefit to replacing the connection until new tokens have been acquired. This release further ensures that infinite memory allocation cannot occur by tracking only the most recent error per fileserver; and only does so if hard mount mode is enabled for the cell.

Processes that issued syscalls which became stuck in a RPC retry loop could not be interrupted. In addition to the unintentional retry loops there are intentional retry loops such as those that occur because a fileserver repeated fails the call with a “VBUSY” error. “VBUSY” originally meant that the required volume was temporarily in use by the volume management service (“volserver”) and would be back in service soon so please wait. Beginning with IBM AFS 3.4 the error code was used to force a retry for reasons having nothing to do with the requested volume’s availability. IBM AFS 3.4 introduced the concept of multi-homed clients which meant that the fileserver could no longer track the identity of a cache manager by its IPv4 address. Instead, each cache manager randomly generates a UUID which the fileserver must query each time an incoming RPC is received on a previously unrecognized Rx connection. An Rx connection might be unrecognized because it’s new, or because the client’s IP address changed, or because the fileserver garbage collected its state information. Regardless of the reason, if a call is received and there is no Rx connection binding to a cache manager UUID, then the fileserver application worker thread to which the call was assigned must issue an outgoing RPC to the client’s RXAFSCB service. Each client has at most one worker thread. Prior to this release, every incoming RXAFSCB call required acquisition of the GLOCK before it could execute. If the GLOCK is unavailable, the RXAFSCB thread blocks and no other pending RXAFSCB calls can be processed. If a RXAFSCB call blocks for too long, the fileserver timeouts trigger and the fileserver replies to the incoming RXAFS call with “VBUSY”. “VBUSY” can also be sent if the fileserver’s quota on the number of calls from the same endpoint is exceeded.

The v2021.05-54 release removes the GLOCK requirement for many RXAFSCB RPCs including all of the UUID query RPCs and all whole volume callback RPCs. Callback RPCs for individual vnodes must still acquire the GLOCK. However, it is hoped that these changes will significantly reduce the time required to process the queries generated due to unrecognized Rx connections. The AuriStor Rx stack permits a dozen calls to be issued on a single logical connection which the fileserver sees as three unique Rx connections from the same endpoint.

Each Unix UID or PAG ID must use an independent Rx connection to each fileserver. On a system which processes /afs syscalls for 1200 UID or PAG IDs in a minute, there is the potential for 3600 Rx connections per fileserver. When the fileserver is OpenAFS, this quickly triggers per endpoint throttling and generates VBUSY storms which can result in unlucky client processes experiencing starvation. With this release, starving processes can be interrupted and RXAFSCB UUID queries can be completed without blocking.

AuriStor believes this release will provide significant benefits for large scale compute nodes, web servers and time sharing systems.

v2021.05-53 (4 January 2025)

v2021.05-52 switched the memory allocation strategy for rx_event objects from a global free list to reliance upon the operating system's memory allocator to allocate individual objects which are freed upon rx_event completion or cancelation. Reliance upon the system allocator introduces the risk of allocation failures due to out of memory conditions. If allocation of an rx_event object fails, a reference count leak on the associated rx_connection or rx_call object can occur preventing garbage collection of the object.
This release handles rx_event allocation failures:
- delayed ACK or delayed ABORT events are replaced by immediate transmission of the ACK or ABORT
- other cases fail the call or connection with an RX_OUT_OF_MEMORY error which is translated on the wire to RX_WIRE_CALL_DEAD (-1).
After executing a call to a fileserver, if the Rx connection is in an error state for any reason other than expired tokens, then schedule the Rx connection to be replaced the next time a call is attempted. All calls issued on an Rx connection that is already in an error state fail immediately preventing further communication with the associated fileserver from the UID/PAG.
Rx connections in an expired token error state are not replaced until new tokens are set.
Rx connections can be placed into an error state either due to local or remote failures. Such failures are often generated by the in-use security class (RXKAD/RXGK) but might be due to an out-of-memory condition, system clock advancement or reversal, or even application errors.
This release includes internal code changes to the management of servers, connections and access rights which will be used by a future release to separate PAG ID allocation from the system's UID namespace.
If 'afsd' is configured with a keytab, receipt of a PoweredOn event will force an attempt to acquire system tokens.
On supported macOS versions, enforce the use of code signing identities to authenticate mach messages involving the AFSBackgrounder and the AFSPreference panel.

v2021.05-52 (5 December 2024)

The v2021.05-50 replacement of the linked list implementation of the vcache access cache with a btree and last lookup cache introduced a memory corruption bug which is fixed in this release.
Eliminate the GLOCK requirement for the collection of function call and RPC statistics collection.
Replace small space allocation with a GLOCK free allocator.
Reimplement the cache manager tracing logging routines so they no longer require the GLOCK.
The Rx networking stack schedules many different types of time sensitive events associated with individual packet retransmission, delayed ack and abort processing, path mtu detection, keep alives, and other categories. Prior to this release, all memory allocated to Rx events objects would not be freed to the operating system until the userspace process terminated or the kernel module was unloaded.
Beginning with this release, instead of having a single, global, free list for rxevents, the system allocator is trusted to be able to allocate memory quickly. This simplifies the Rx implementation, and should be more efficient, as most system allocators will have per thread or core slabs, removing the contention of the global lock protecting the rxevent free list. In addition, the change protects against a network event allocating memory that cannot be readily freed.
On non-Total Store Ordering (TSO)architectures, relaxed atomics used as reference counters do not ensure completion of all prior stores or recency of later loads. This release replaces all read-modify-write atomics to be acquire-release, which, among other things, ensures that only the intended ordering is possible.

v2021.05-51 (26 November 2024)

Fix a vcache (aka inode) reference leak that can occur during an access rights query for a non-directory object if the neither the calling process nor the cache manager possesses tokens for the cell containing the object. The leak was introduced in v2021.05-45.
When scanning the token list for each user searching for expired tokens the user spinlock must be held to prevent races. The updated user management implementation pulled into v2021.05-45 failed to hold the spinlock.
During cache manager module shutdown reset global structures to the module initialization state. Doing so improves the likelihood that the module can be reinitialized.

v2021.05-50 (23 November 2024)

The linked list implementation of the vcache access rights cache was replaced with a btree and a last lookup cache. The btree design is more efficient for lookups and removals, and scales better as the number of uids and/or process authentication groups (pags) accessing /afs increases.
Prune access rights for expired users. Doing so reduces the cost of subsequent lookups.
Cache the access rights obtained for a uid|pag via the "anyAccess" field. Previously, a uid|pag that obtained its rights from the "anyAccess" field had to perform and fail a rights lookup on each and every vcache access.
When setting or clearing tokens for a single cell, no longer invalidate the access rights cached for the uid|pag for all cells. Instead, only invalidate the access rights cached for the cell whose tokens were set or cleared.

v2021.05-49 (16 November 2024)

The output of "tokens" command failed to report yfs-rxgk tokens was broken starting in v2021.05-46

v2021.05-48 (12 November 2024)

Preallocated buffer overflows in XDR responses (CVE-2024-10397)
The AuriStorFS and AFS3 RPC suites rely upon Sun RPC XDR to marshal binary data structures for network transfer. The AuriStor XDR implementation is derived from Sun Microsystems' Sun RPC code base. The Sun RPC XDR API permits memory for output parameters to (optionally) be preallocated which can result in various classes of memory corruption and/or memory leaks in RPC initiator processes.
The AuriStorFS v2021.05-48 release introduces additional data length validation checks within the AuriStor XDR implementation and prohibits the use of preallocated memory for string output parameters or fields. All cache managers, servers and command line tools are modified by these changes.

v2021.05-46 (28 October 2024)

Cache Manager:
- Prevent a kernel memory leak when server preferences are set via the yfs-client.conf [afsd] configuration or via "fs setserverprefs".
- Directory enumeration of a truncated directory now returns an error instead of assuming the end of the directory has been reached.
- Since AFS 3.0, the Unix cache manager has used the root identity credentials to create anonymous outgoing connections to the location service and each fileserver. However, if uid 0 is assigned a token, then those Rx connections will no longer be anonymous. Beginning with this release anonymous outgoing connections are always created with the NOPAG identity (uid 0xffffffff) instead of the root identity.
- When establishing an outgoing rxgk connection, do not fallback to the systemuser's credentials if the user's credentials resulted in a fatal error. Falling back to the systemuser's credentials can result in inappropriate use of an anonymous connection.
- Improved access rights cache correctness for YFS servers
  In prior releases, the access check logic used the file rights for any files fetched from an AuriStorFS fileserver. For files fetched from an AFS-3 fileserver (and, historically, for all files), it used the directory rights, with the (a)dmin right from the file mixed in. The (a)dmin right on a non-directory indicates that the object is owned by the authenticated user.
  This approach has some issues when combined with the access rights cache, and current fileserver callback behavior. On an AuriStorFS file server, the rights on a non-directory may be determined by the rights granted on its parent directory or, with per-file ACLs, those granted on the object itself. The fileserver will only break a non-directory's callback when a per-file ACL is changed - changing a directory ACL will not break callbacks on files within that directory. This means that changing a directory ACL will not invalidate access rights cache entries on files in tha directory, even if the effective ACL on this files has changed, and the cached rights are no longer correct.
  This release works around this by adding a new function which returns the access rights for a file hosted on an AuriStor fileserver. It uses the parent vnode information to locate the parent directory. If the parent directory isn't in the cache, or it doesn't have a valid callback, or if it has been changed since the file's access rights were cached, it clears the current access rights. Files without a parent directory must have per-file ACLs, and so their cached rights can be safely used.
  Note that files with parent vnodes may still have per-file ACLs, and that the breadcrumbing performed by the client may add parent vnode fields to vnodes which don't have them provided by the fileserver. Such vnodes may have their cached access rights cleared more frequently than necessary.
- Add a new mechanism for caching access rights within the vcache structure. This cache is protected via a vcache specific spinlock, and can be accessed without holding the GLOCK.
  This new cache mechanism returns the memory associated with cached rights back to the kernel's slab free memory pool instead of adding the unused rights structures to a cache manager managed free list. The previous cache implementation never returned allocated memory to the kernel. Instead, invalidated access rights were appended to a free access rights queue for later reuse.
- When a volume is accessed via multiple mountpoints a choice must be made regarding which mountpoint is considered to be the active (or parent) mountpoint. This release alters the behavior such that the active mountpoint is set every time a mountpoint is traversed.
  This behavior is easier to understand and is more likely to provide the expected result for a single process that repeatedly accesses volumes from multiple mountpoints. However, it can result in unexpected results when multiple processes are traversing multiple mountpoints in parallel without any synchronization.

v2021.05-45 (Not released)

v2021.05-44a (18 September 2024)

Authentication:
- AuriStorFS v2021.05-44 included an updated version of the Heimdal Kerberos framework used by AuriStorFS when acquiring yfs-rxgk and rxkad authentication tokens. The updated Heimdal included a bug which disabled the use of DNS SRV records for KDC discovery and DNS TXT records for realm discovery. As a side effect token acquisition might fail with an unable to reach any KDC in realm error. This is fixed in v2021.05-44a.

v2021.05-44 (17 August 2024)

Cache Manager:
- Since v0.192 the cache manager has failed acquire the global lock when upgrading a shared-lock to a write-lock during the execution of a background cache chunk file truncation.
Authentication:
- Neither MIT nor Heimdal gssapi nor their gss mechanisms consistently initialize the output 'minorStatus' parameter. Various functions can return either success or failure majorStatus values with minorStatus unassigned. As a result, stack garbage will be used when generating error messages. From now on libyfs_acquire will always initialize the minorStatus output variable to zero before calling into the gssapi library.
Command Parser:
- No longer accept the token "-" as a switch which eventually fails with a CMD_UNKNOWNSWITCH error. Instead, process the token as a data value.
- Optimize the processing of the loop which processes "source" command input.
- If the source command input file is "-", read from stdin.

v2021.05-41 (26 June 2024)

Rx Networking (libyfs_rx):
- A race during event creation can lead to the freeing of the event while its still in use.
- RFC1122 says that Net and Host unreachable ICMP errors might be transient and should therefore not be treated as fatal. There is no such language for the equivalent ICMPV6 errors. However, in practice ICMP6_DST_UNREACH_NOROUTE, ICMP6_DST_UNREACH_BEYONDSCOPE, and ICMP6_DST_UNREACH_ADDR can be transient.
  Linux has considered these ICMPV6 destination unreachable errors as non-fatal going back at least as far as the initial git repository commit.
  AuriStor Rx has always treated these as fatal errors which results in immediate termination of in-flight calls when received. Even if the network route corrects itself before the call timeout period expires. This release mirrors the Linux behavior and makes these errors non-fatal.
Cache Manager:
- For the first time the cache manager can detect the deletion of a volume and handle the creation of a new volume with the same name but a different volume id.
- If the location service reports the deletion of a volume, invalidate all mount points to that volume.
- RXAFS_GetCapabilities RPC failures should not be treated as a fatal error preventing failover to another replica site.
Authentication ("libyfs_acquire") used by aklog, vos, pts, bos, afsio:
- rxkad_k5 token acquisition krb5 ccache management. This release altered the krb5 credential cache management strategy once again to work around different bugs in MIT krb5 and Heimdal.
- New ACQUIRE_ERR_CRED_EXPIRED error code introduced to represent the case when a request for a service credential returns one that is already expired.
Command parser (libyfs_cmd):
- When parsing configuration files there is a depth limit of ten active inclusions. This limit was improperly enforced as a limit of ten included files instead of a depth of ten included files. As of this release it is now possible to populate an includedir directory with any number of .conf files.

v2021.05-40

Not released.

v2021.05-39 (20 May 2024)

Parallel Random Number Generation:
- AuriStorFS processes rely upon the krb5_generate_random() and RAND_bytes() functions to obtain random bytes for cryptographic operations and random counters. krb5_generate_random() internally acquires a mutex to protect internal state information. This mutex has become a significant barrier to the encryption and checksumming of Rx packets with both yfs-rxgk and rxkad.
  
  This release replaces general use of krb5_generate_random() and RAND_bytes() with a per-thread ChaCha20 CS-PRNG. This avoids the acquisition of a global mutex and permits increased parallelism on multi-core systems.
Rx Networking (libyfs_rx):
- The Rx network stack schedules a garbage collection operation to execute once per minute. This operation enforces call timeouts, destroys idle connections and destroys idle peers. The operation has historically been performed by the Rx event thread which is already responsible for performing actions in response to call RTOs, sending NAT Ping and keep-alive packets, and retrying connection challenge and reachability checks.
  
  The time complexity of the garbage collection operation is determined by the number of calls, connections, and peers. The busier the Rx endpoint the more work must be performed during each garbage collection run and the longer it takes to complete. While garbage collection is active other events cannot be processed which can interfere with the proper flow control of active calls.
  
  As with all Rx events, the garbage collection event is scheduled to execute at an absolute clock time. If the system clock drifts (or is administratively set) backwards garbage collection will not be performed until the clock catches up with the scheduled time.
  
  Another responsibility of the garbage collection procedure is to terminate calls if the system clock drifted backwards by five minutes or longer. However, when the clocked drifts backwards garbage collection is not performed until the clock has advanced beyond the point where calls require termination. As a result, calls are not terminated due to backwards clock drift and they can stall.
  
  This release re-implements the garbage collection procedure using a dedicated thread and relative waits. This change ensures that the garbage collection procedure will not prevent the execution of call related events and permits calls to be terminated when large backward clock drifts are detected.
Disk Cache Management:
- Since IBM AFS 3.5, the cache has been considered "too full" even if there exist cache files have been discarded but not yet truncated. When the cache is "too full" most operations that write to the cache will block until truncation of discarded cache files has been performed which results in unnecessary delays. This release fixes the cache such that that discarded but not yet truncated cache files do not block write operations.
- This release permits the cache truncation daemon thread to exit sooner if the cache manager is shutting down.
Improved failover when the RXGK service (co-located with each vlserver) fails to issue tokens. The failures might be the result of misconfiguration, an inability to read keys or loss of Ubik quorum.

v2021.05-38 (29 February 2024)

As with other AuriStorFS releases since the beginning of 2024, this release includes additional improvements to the Rx RPC implementation which are related to the possibility of silent data corruption when Rx jumbograms are in use. Prior releases disabled the negotiation of Rx jumbograms such that the v2021.05-37 Rx peer will refuse to send Rx jumbograms and will request that the remote peer does not send them. However, a bad actor could choose to send Rx jumbograms even though they were asked not to. v2021.05-38 introduces additional protections to ensure that a corrupt Rx jumbogram is dropped instead of being accepted.

The v2021.05-38 Rx RPC implementation also includes two optimizations. First, when Rx initiators complete a call they will no longer send an extra ACK packet to the Rx acceptor of the completed call. The sending of this unnecessary ACK creates additional work for the server which can result in increased latency for other calls being processed by the server.

Second, all AuriStor Rx services require a reach check for incoming calls from Rx peers to help protect against Distributed Reflection Denial of Service (DRDoS) attacks and execution of RPCs when the response cannot be delivered to the caller. A new reach check is required for each new call that arrives more than 60 seconds after the prior reach check completed. v2021.05-38 Rx considers the successful acknowledgment of a response DATA packet as a reach check validation. With this change reach checks will not be periodically required for a peer that completes at least one call per 60 seconds. A 1 RTT delay is therefore avoided each time a reach check can be avoided. In addition, reach checks require the service to process an additional ACK packet. Eliminating a large number of reach checks can improve overall service performance.

The final Rx RPC change in this release is specific to kernel implementations. Prior releases restricted the frequency of executing time scheduled Rx events to a granularity no smaller than 500ms. As a result an RTO timer event for a lost packet could not be shorter than 500ms even if the measured RTT for the connection is significantly smaller. The minimum RTO for a connection in AuriStor Rx is 200ms. The inability to schedule shorter timeouts impacts recovery from packet loss.

v2021.05-37 (5 February 2024)

Rx improvements:
- The v2021.05-36 release permanently disabled all use of Rx jumbograms due to a risk of silent data corruption. However, when advertising the number of acceptable datagrams in the ACK trailer a missing htonl() set the value to 16777216 instead of 1 on little-endian systems.
- When sending a PING ACK as a reachability test, ensure that the previousPacket field is properly assigned to the largest accepted DATA packet sequence number instead of zero.
- Replace the initialization state flag with two flags. One that indicates that Rx initialization began and the other that it succeeded. The first prevents multiple attempts at initialization after failure. The second prevents shutdown from accessing uninitialized structures if initialization failed.
Cache Manager Improvements:
- No longer refuse to start if both the 'cachedir' and 'memcache' options are present in the configuration file.
- New variable to store the maximum number of cache blocks used. which is accessible via /proc/fs/auristorfs/cache/blocks_used_max.

v2021.05-36 (10 January 2024)

Rx improvements:
- Permanently disable all use of Rx jumbograms due to a risk of silent data corruption.
  
  Ever since OpenAFS 1.0, and possibly before, a race condition has existed when Rx transmits packets. As the rx_call.lock is dropped when starting packet transmission, there is no protection for data that is being copied into the kernel by sendmsg(). It is critical that this packet data is not modified by another thread. However, races exist between the application, listener, and event threads that can lead to retransmissions starting whilst an original transmission is still in progress. This can lead to the packet headers being overwritten, and either the original transmission, the retransmission or both sending corrupt data to the peer.
  
  This corruption can affect the packet serial number or packet flags. It is particularly harmful when the packet flags are corrupted, as this can lead to multiple Rx packets which were intended to be sent as Rx jumbograms being delivered and misinterpreted as a single large packet. The eventual result of this depends on the Rx security class in play, but it can cause decrypt integrity errors (rxgk:crypt and rxgk:auth) or corruption of the data stream (rxnull, rxgk:clear or rxkad:auth).
  
  All AuriStorFS servers, OpenAFS 1.6 or later servers, and the Windows cache manager have been shipped with Rx jumbograms disabled by default. The UNIX cache managers however are shipped with jumbograms enabled. There are many AFS cells around the world that continue to deploy OpenAFS 1.4 or earlier fileservers which continue to negotiate the use of Rx jumbograms.
  
  It is worth noting that all AuriStorFS v0.198 and later fileservers and cache managers implement explicit checks that will recognize the corrupted application data stream and prevent corrupted file content from being stored either into an AFS vnode's backing store in the volume's object store or the cache manager's AFS cache. OpenAFS cache managers and fileservers do not have these protections.
  
  With Rx jumbograms disabled the maximum number of Rx packets in a datagram is reduced from 6 to 1; the maximum number of send and receive datagram fragments is reduced from 4 to 1; and the maximum advertised MTU is restricted to 1444 - the maximum rx packet size prior to the introduction of jumbograms in IBM AFS 3.5.
- If the rx call flow state transitions from either the RECOVERY or RESCUE states to the LOSS state as a result of an RTO resend event while writing packets to the network, cease transmission of any new DATA packets if there are packets in the resend queue.
- When the call flow state is LOSS and all packets in the resend queue have been retransmitted and yet the recovery point has not been reached, then permit new DATA packets to be sent in order to maintain a full congestion window.
- Add a safety check to prevent the estimated RTT from underflowing when the actual roundtrip time is smaller than 125us.
- Fix the computation of the padding required for rxgk encrypted packets. This bug results in packets sending 8 bytes fewer per packets than the network permits. This bug accidentally prevented the construction of Rx jumbograms when a call is protected by rxgk:crypt.
- Replace the random number generator with a more security source of random bytes.

v2021.05-33 (27 November 2023)

Rx improvements:
- Not all calls transfer enough data to be able to measure a smoothed round-trip time (SRTT). Calls which are unable to compute a SRTT should not be used to update the peer host RTO value which is used to initialize the RTO for subsequent calls.
  
  Without this change, a single DATA packet call will cause the peer host RTO to be reduced to 0ms. Subsequent calls will start with a RTO value of MAX(0, rxi_minPeerTimeout) where rxi_minPeerTimeout defaults to 200ms. If the actual measured RTO is greater than 200ms, then initial RTO will be too small resulting in premature triggering of the RTO timer and the call flow state entering the loss phase which can significantly hurt performance.
- Initialize the peer host RTO to rxi_minPeerTimeout (which defaults to 200ms) instead of one second. Although RFC6298 recommends the use of one second when no SRTT is available, Rx has long used the rxi_minPeerTimeout value for other purposes which are supposed to be consistent with initial RTO value. It should be noted that Linux TCP uses 200ms instead of one second for this purpose.
- If associating a security class with an Rx connection fails immediately place the Rx connection into an error state. A failure might occur if the security class is unable to access valid key material.
- If an incoming Rx call requires authentication and the security class is unable to successfully generate a challenge, put the incoming Rx connection into an error state and issue an abort to the caller.
- If an incoming Rx call requires authentication and the security class is able to generate a challenge but the challenge cannot be returned to Rx, then treat this as a transient error. Do not acknowledge the incoming DATA packet and do not place the Rx connection into an error state. An attempt to re-issue the challenge will be performed when the DATA packet is retransmitted.
- If an Rx call is terminated due to the expiration of the configured connection dead time, idle dead time, hard dead time, or as a result of clock drift, then send an ABORT to the peer notifying them that the call has been terminated. This is particularly important for terminated outgoing calls. If the peer does not know to terminate the call, then the call channel might be in use when the next outgoing call is issued using the same call channel. If the next incoming call is received by an in-use call channel, the receiver must drop the received DATA packet and return a BUSY packet. The call initiator will need to wait for a retransmission timeout to pass before retransmitting the DATA packet. Receipt of BUSY packets cannot be used to keep a call alive and therefore the requested call is at greater risk of timing out if the network path is congested.
aklog and krb5.log (via libyfs_acquire):
- If the linked Kerberos library implements krb5_cc_cache_match() and libacquire has been told to use an explicit principal name and credential cache, the Kerberos library might return KRB5_CC_NOTFOUND even though the requested credential cache is the correct one to use. This release will not call krb5_cc_cache_match() if the requested credential cache contains the requested principal.
Cell Service Database (cellservdb.conf):
- cellservdb.conf has been synchronized with the 31 Oct 2023 update to the grand.central.org CellServDB file.

v2021.05-32 (9 October 2023)

No significant changes for macOS compared to v2021.05-31

v2021.05-31 (25 September 2023)

New platform:
- macOS 14 Sonoma
macOS 14 Sonoma:

AuriStorFS v2021.05-29 and later installers for macOS 13 Ventura are compatible with macOS 14 Sonoma and do not need to be removed before upgrading to macOS 14 Sonoma. Installation of the macOS 14 Sonoma version of AuriStorFS is recommended.

Cache Manager:
- If an AuriStorFS cache manager is unable to use the yfs-rxgk security class when communicating with an AuriStorFS fileserver, it must assume it is IBM AFS 3.6 or OpenAFS and upgrade it to AuriStorFS if an upgrade probe returns a positive result. Once a fileserver's type is identified as AuriStorFS the type should never be reset; even if communication with the fileserver is lost or the fileserver restarts.
  
  If an AuriStorFS fileserver is replaced by an OpenAFS fileserver on the same endpoint, then the UUID of the OpenAFS must be different. As a result, the OpenAFS fileserver will be observed as distinct from the AuriStorFS fileserver that previously shared the endpoint.
  
  Prior to this release there were circumstances in which the cache manager discarded the fileserver type information and would fail to recognize the fileserver as an AuriStorFS fileserver when yfs-rxgk could not be used. This release prevents the cache manager from resetting the type information if the fileserver is marked down.
- If a fileserver's location service entry is updated with a new uniquifier value (aka version number), this indicates that one of the following might have changed:
  1. the fileserver's capabilities
  2. the fileserver's security policy
  3. the fileserver's knowledge of the cell-wide yfs-rxgk key
  4. the fileserver's endpoints
  Beginning with this release the cache manager will force the establishment of new Rx connections to the fileserver when the uniquifier changes. This ensures that the cache manager will attempt to fetch new per-fileserver yfs-rxgk tokens from the cell's RXGK service, enforce the latest security policy, and not end up in a situation where its existing tokens cannot be used to communicate with the fileserver.
aklog:
- Fix incorrect output when populating the server list for a service fails. The stashed extended error explaining the cause of the failure was not displayed.
- If a cell has neither _afs3-prserver._udp. DNS SRV records nor AFSDB records, the lookup of the cell's protection servers would fail if there is no local cell configuration details. The fallback to use _afs3-vlserver._udp. DNS SRV records did not work. This is corrected in this release.

v2021.05-30 (6 September 2023)

Do not mark a fileserver down in response to a KRB5 error code.
fs cleanacl must not store back to the file server a cleaned acl if it was inherited from a directory. Doing so will create a file acl.
Correct the generation of never expire rxkad_krb5 tokens from Kerberos v5 tickets which must have a start time of Unix epoch and an end time of 0xFFFFFFFF seconds. The incorrectly generated tokens were subject to the maximum lifetime of 30 days.
Correct the generation of the yfs-rxgk RESPONSE packet header which failed to specify the key version generation number used to encrypt the authenticator. If the actual key version is greater than zero, then the authenticator would fail to verify.
Enforce a maximum NAT ping period of 20s to ensure that NAT/PAT/firewall rules due not expire while Rx RPCs are in-flight.

v2021.05-29 (26 June 2023)

Execution of fs commands such as examine, whereis, listquota, fetchacl, cleanacl, storeacl, whoami, lsmount, bypassthreshold and getserverprefs could result in memory leaks by the AuriStorFS kernel extension.

v2021.05-27 (1 May 2023)

Fixes for bugs in vos introduced in v2021.05-26.

v2021.05-26 (17 April 2023)

Fixed a potential kernel memory leak when triggered by fs examine, fs listquota, or fs quota.
Increased logging of VBUSY, VOFFLINE, VSALVAGE, and RX_RESTARTING error responses. A log message is now generated whenever a task begins to wait as a result of one of these error responses from a fileserver. Previously, a message was only logged if the volume location information was expired or discarded.
Several changes to optimize internal volume lookups.
Faster failover to replica sites when a fileserver returns RX_RESTARTING, VNOVOL or VMOVED.
rxdebug regains the ability to report rx call flags and rx_connection flags.
The RXRPC library now terminates calls in the QUEUED state when an ABORT packet is received. This clears the call channel making it available to accept another call and reduces the work load on the worker thread pool.
Fileserver endpoint registration changes no longer result in local invalidation of callbacks from that server.
Receipt of an RXAFSCB_InitCallBackState3 RPC from a fileserver no longer resets the volume site status information for all volumes on all servers.

v2021.05-25 (28 December 2022)

The v2021.05-25 release includes further changes to RXRPC to improve reliability. The changes in this release prevent improper packet size growth. Packet size growth should never occur when a call is attempting to recover from packet loss; and is unsafe when the network path's maximum transmission unit is unknown. Packet size growth with be re-enabled in a future AuriStorFS release that includes Path MTU detection and the Extended SACK functionality.
Improved error text describing the source of invalid values in /etc/yfs/yfs-client.conf or included files and directories.

v2021.05-24 (25 October 2022)

New Platform: macOS 13 (Ventura)

v2021.05-22 (12 September 2022) and v2021.05-21 (6 September 2022)

RX RPC
- If receipt of a DATA packet causes an RX call to enter an error state, do not send the ACK of the DATA packet following the ABORT packet. Only send the ABORT packet.
- AuriStor RX has failed to count and report the number of RX BUSY packets that have been sent. Beginning with this change the sent RX BUSY packet count is once again included in the statistics retrieved via rxdebug server port -rxstats.
- Introduce minimum and maximum bounds checks on the ACK packet trailer fields. If the advertised values are out of bounds for the receiving RX stack, do not abort the call but adjust the values to be consistent with the local RX RPC implementation limits. These changes are necessary to handle broken RX RPC implementations or prevent manipulation by attackers.

New to v2021.05-20 (15 August 2022) and v2021.05-19 (13 August 2022)

RX RPC
- Include the DATA packet serial number in the transmitted reachability check PING ACK. This permits the reachability test ACK to be used for RTT measurement.
- Do not terminate a call due to an idle dead timeout if there is data pending in the receive queue when the timeout period expires. Instead deliver the received data to the application. This change prevents idle dead timeouts on slow lossy network paths.
- Fix assignment of RX DATA, CHALLENGE, and RESPONSE packet serial numbers in macOS (KERNEL). Due to a mistake in the implementation of atomic_add_and_read the wrong serial numbers were assigned to outgoing packets.

New to v2021.05-18 (12 July 2022)

Cache Manager
- Prevent a kernel memory leak of less than 64 bytes for each bulkstat RPC issued to a fileserver. Bulkstat RPCs can be frequently issued and over time this small leak can consume a large amount of kernel memory. Leak introduced in AuriStorFS v0.196.
- The Perl::AFS module directly executes pioctls via the OpenAFS compatibility pioctl interface instead of the AuriStorFS pioctl interface. When Perl::AFS is used to store an access control list (ACL), the deprecated RXAFS_StoreACL RPC would be used in place of the newer RXAFS_StoreACL2 or RXYFS_StoreOpaqueACL2 RPCs. This release alters the behavior of the cache manager to use the newer RPCs if available on the fileserver and fallback to the deprecated RPC. The use of the deprecated RPC was restricted to use of the OpenAFS pioctl interface.
RX RPC
- Handle a race during RX connection pool probes that could have resulted in the wrong RX Service ID being returned for a contacted service. Failure to identify that correct service id can result in a degradation of service.
- The Path MTU detection logic sends padded PING ACK packets and requests a PING_RESPONSE ACK be sent if received. This permits the sender of the PING to probe the maximum transmission unit of the path. Under some circumstances attempts were made to send negative padding which resulted in a failure when sending the PING ACK. As a result, the Path MTU could not be measured. This release prevents the use of negative padding.
Preparation for supporting macOS 13 Ventura when it is released in Fall 2022.
Some shells append a slash to an expanded directory name in response to tab completion. These trailing slashes interfered with "fs lsmount", "fs flushmount" and "fs removeacl" processing. This release includes a change to prevent these commands from breaking when presented a trailing slash.

New to v2021.05-17 (16 May 2022)

Cell Service Database Updates
- Update cern.ch, ics.muni.cz, ifh.de, cs.cmu.edu, qatar.cmu.edu, it.kth.se
- Remove uni-hohenheim.de, rz-uni-jena.de, mathematik.uni-stuttgart.de, stud.mathematik.uni-stuttgart.de, wam.umd.edu
- Add ee.cooper.edu
- Restore ams.cern.ch, md.kth.se, italia
Fix parsing of [afsd] rxwindow configuration which can be used to specified a non-default send/receive RX window size. The current default is 128 packets.
RX Updates
- Add nPacketsReflected

Logging improvements

Cache directory validation errors log messages now include the cache directory path.

Log the active configuration path if "debug" logging is enabled.

More details of rxgk token extraction failures.

New to v2021.05-16 (24 March 2022)

RX - Previous releases re-armed the Retransmission Timeout (RTO) each time a new unacknowledged packet is acknowledged instead of when a new leading edge packet is acknowledged. If leading edge data packet and its retransmission are lost, the call can remain in the "recovery" state where it continues to send new data packets until one of the following is true:
. the maximum window size is reached
. the number of lost and resent packets equals 'cwind'
at which point there is nothing left to transmit. The leading edge data packet can only be retransmitted when entering the "loss" state but since the RTO is reset with each acknowledged packet the call stalls for one RTO period after the last transmitted data packet is acknowledged.

This poor behavior is less noticiable with small window sizes and short lived calls. However, as window sizes and round-trip times increase the impact of a twice lost packet becomes significant.

RX - Never set the high-order bit of the Connection Epoch field. RX peers starting with IBM AFS 3.1b through AuriStor RX v0.191 ignore the source endpoint when matching incoming packets to RX connections if the high-order epoch bit is set. Ignoring the source endpoint is problematic because it can result in a call entering a zombie state whereby all PING ACK packets are immediately responded to the source endpoint of the PING ACK but any delayed ACK or DATA packets are sent to the endpoint bound to the RX connection. An RX client that moves from one network to another or which has a NAT|PAT device between it and the service can find themselves stuck.

Starting with AuriStor RX v0.192 the high-order bit is ignored by AuriStor RX peer when receiving packets. This change to always clear the bit prevents IBM AFS and OpenAFS peers from ignoring the source endpoint.

RX - The initial packetSize calculation for a call is altered to require that all constructed packets before the receipt of the first ACK packet are eligible for use in jumbograms if and only if the local RX stack has jumbograms enabled and the maximum MTU is large enough. By default jumbograms are disabled for all AuriStorFS services. This change will have a beneficial impact if jumbograms are enabled via configuration; or when testing RX performance with "rxperf".

New fs whereis -noresolve option displays the fileservers by network endpoint instead of DNS PTR record hostname.

New to v2021.05-15 (24 January 2022)

kernel - fixed YFS_RXGK service rx connection pool leak

New to v2021.05-14 (20 January 2022)

fs mkmount permit mount point target strings longer than 63 characters.
afsd enhance logging of yfs-rxgk token renewal errors.

afsd gains a "principal = configuration option for use with keytab acquisition of yfs-rxgk tokens for the cache manager identity.

kernel - Avoid unnecessary rx connection replacement by racing threads after token replacement or expiration.

kernel - Fix a regression introduced in v2021.05 where an anonymous combined identity yfs-rxgk token would be replaced after three minutes resulting in the connection switching from yfs-rxgk to rxnull.

kernel - Fix a regression introduced in v0.208 which prevented the invalidation of cached access rights in response to a fileserver callback rpc. The cache would be updated after the first FetchStatus rpc after invalidation.

kernel - Reset combined identity yfs-rxgk tokens when the system token is replaced.

kernel - The replacement of rx connection bundles in the cache manager to permit more than four simultaneous rx calls per uid/pag with trunked rx connections introduced the following regressions in v2021.05.

a memory leak of discarded rx connection objects

failure of NAT ping probes after replacement of an connection

inappropriate use of rx connections after a service upgrade failure

All of these regressions are fixed in patch 14.

New to v2021.05-12 (7 October 2021)

fs ignorelist -type afsmountdir in prior releases could prevent access to /afs.

Location server rpc timeout restored to two minutes instead of twenty minutes.

Location server reachability probe timeout restored to six seconds instead of fifty seconds.

Cell location server upcall results are now cached for fifteen seconds.

Multiple kernel threads waiting for updated cell location server reachability probes now share the results of a single probe.

RX RPC implementation lock hierarchy modified to prevent a lock inversion.

RX RPC client connection reference count leak fixed.

RX RPC deadlock during failed connection service upgrade attempt fixed..

New to v2021.05-9 (25 October 2021)

First public release for macOS 12 Monterey build using XCode 13. When upgrading macOS to Monterey from earlier macOS releases, please upgrade AuriStorFS to v2021.05-9 on the starting macOS release, upgrade to Monterey and then install the Monterey specific v2021.05-9 release.

Improved logging of "afsd" shutdown when "debug" mode is enabled.

Minor RX network stack improvements

New to v2021.05-3 (10 June 2021)

Fix for [cells] cellname = {...} without server list.

New to v2021.05 (31 May 2021)

Multi-homed location servers are finally managed as a single server instead of treating each endpoint as a separate server. The new functionality is a part of the wholesale replacement of the former cell management infrastructure. Location server communication is now entirely managed as a cluster of multi-homed servers for each cell. The new infrastructure does not rely upon the global lock for thread safety.

This release introduces a new infrastructure for managing user/pag entities and tracking their per cell tokens and related connection pools.

Expired tokens are no longer immediately deleted so that its possible for them to be listed by "tokens" for up to two hours.

Prevent a lock inversion introduced in v0.208 that can result in a deadlock involving the GLOCK and the rx call.lock. The deadlock can occur if a cell's list of location servers expires and during the rebuild an rx abort is issued.

Add support for rxkad "auth" mode rx connections in addition to "clear" and "crypt". "auth" mode provides integrity protection without privacy.

Add support for yfs-rxgk "clear" and "auth" rx connection modes.

Do not leak a directory buffer page reference when populating a directory page fails.

Re-initialize state when populating a disk cache entry using the fast path fails and a retry is performed using the slow path. If the data version changes between the attempts it is possible for truncated disk cache data to be treated as valid.

Log warnings if a directory lookup operation fails with an EIO error. An EIO error indicates that an invalid directory header, page header, or directory entry was found.

Do not overwrite RX errors with local errors during Direct-I/O and StoreMini operations. Doing so can result in loss of VBUSY, VOFFLINE, UAENOSPC, and similar errors.

Correct a direct i/o code path which could overwrite a fileserver returned error code with a local RXGEN_CC_UNMARSHAL error.

Correct the StoreMini code path which could overwrite a fileserver returned error code with a local RXGEN_CC_UNMARSHAL error.

Ensure the rx call object is not locked when writing to the network socket.

Removed all knowledge of the KERNEL global lock from RX. Acquiring the GLOCK from RX is never safe if any other lock is held. Doing so is a lock order violation that can result in deadlocks.

Fixed a race in the opr_reservation system that could produce a cache entry reference undercount.

If a directory hash chain contains a circular link, a buffer page reference could be leaked for each traversal.

Each AFS3 directory header and page header contains a magic tag value that can be used in a consistency check but was not previously checked before use of each header. If the header memory is zero filled during a lookup, the search would fail producing an ENOENT error. Starting with this release the magic tag values are validated on each use. An EIO error is returned if there is a tag mismatch.

"fs setcrypt -crypt auth" is now a permitted value. The "auth" mode provides integrity protection but no privacy protection.

Add new "aklog -levels " option which permits requesting "clear" and "auth" modes for use with yfs-rxgk.

Update MKShim to Apple OpenSource MITKerberosShim-79.

Report KLL errors via a notification instead of throwing an exception which (if not caught) will result in process termination.

If an exception occurs while executing "unlog" catch it and ignore it. Otherwise, the process will terminate.

New to v2021.04 (22 April 2021)

Primarily bug fixes for issues that have been present for years.

A possibility of an infinite kernel loop if a rare file write / truncate pattern occurs.

A bug in silly rename handling that can prevent cache manager initiated garbage collection of vnodes.

New to v0.209 (13 March 2021)

fs setserverprefs and fs getserverprefs updated to support IPv6 and CIDR specifications.

Improved error handling during fetch data and store data operations.

Prevents a race between two vfs operations on the same directory which can result in caching of out of date directory contents.

Use cached mount point target information instead of evaluating the mount point's target upon each access.

Avoid rare data cache thrashing condition.

Prevent infinite loop if a disk cache error occurs after the first page in a chunk is written.

Network errors are supposed to be returned to userspace as ETIMEDOUT. Previously some were returned as EIO.

When authentication tokens expire, reissue the fileserver request anonymously. If the anonymous user does not have permission either EACCES or EPERM will be returned as the error to userspace. Previously the vfs request would fail with an RXKADEXPIRED or RXGKEXPIRED error.

If growth of an existing connection vector fails, wait on a call slot in a previously created connection instead of failing the vfs request.

Volume and fileserver location query infrastructure has been replaced with a new modern implementation.

Replace the cache manager's token management infrastructure with a new modern implementation.

New to v0.206 (12 January 2021) - Bug fixes

Prevents a possible panic during unmount of /afs.

Improved failover and retry logic for offline volumes.

New to v0.205 (24 December 2020) - Bug fixes

Volume name-to-id cache improvements

Fix expiration of name-to-id cache entries

Control volume name-to-id via sysctl

Query volume name-to-id statistics via sysctl

Improve error handling for offline volumes

Fix installer to prevent unnecessary installation of Rosetta 2 on Apple Silicon

New to v0.204 (25 November 2020) - Bug fix for macOS Big Sur

v0.204 prevents a kernel panic on Big Sur when AuriStorFS is stopped and restarted without an operating system reboot.

introduces a volume name-to-id cache independent of the volume location cache.

New to v0.203 (13 November 2020) - Bug fix for macOS

v0.203 prevents a potential kernel panic due to network error.

New to v0.201 (12 November 2020) - Universal Big Sur (11.0) release for Apple Silicon and Intel

v0.201 introduces a new cache manager architecture on all macOS versions except for High Sierra (10.12). The new architecture includes a redesign of:

kernel extension load

kernel extension unload (not available on Big Sur)

/afs mount

/afs unmount

userspace networking

The conversion to userspace networking will have two user visible impacts for end users:
The Apple Firewall as configured by System Preferences -> Security & Privacy -> Firewall is now enforced. The "Automatically allow downloaded signed software to receive incoming connections" includes AuriStorFS.

Observed network throughput is likely to vary compared to previous releases.

On Catalina the "Legacy Kernel Extension" warnings that were displayed after boot with previous releases of AuriStorFS are no longer presented with v0.201.

AuriStorFS /afs access is expected to continue to function when upgrading from Mojave or Catalina to Big Sur. However, as AuriStorFS is built specifically for each macOS release, it is recommended that end users install a Big Sur specific AuriStorFS package.
AuriStorFS on Apple Silicon supports hardware accelerated aes256-cts-hmac-sha1-96 and aes128-cts-hmac-sha1-96 using AuriStor's proprietary implementation.

New to v0.200 (4 November 2020) - Final release for macOS El Capitan (10.11)

The network path between a client and a server often traverses one or more network segments separated by NAT/PAT devices. If a NAT/PAT times out a RPCs endpoint translation mid-call, this can result in an extended delay before failure and the server being marked down, or worse, a call that never terminates and a client that appears to hang until the fileserver is restarted.

This release includes significant changes to the RX stack and the UNIX cache manager to detect such conditions, fail the calls quickly and detect when it is safe to retry the RPC.

NAT/PAT devices that drop endpoint mappings while in use are anti-social and can result in unwanted delays and even data loss. Whenever possible they should be avoided when they can be. That said, the changes in this release are a huge step to making the loss of endpoint mappings tolerable.

Fix segmentation fault of Backgrounder when krb5_get_credentials() fails due to lack of network connectivity.

Fix the "afsd" rxbind option which was ignored if the default port, 7001, is in use by another process on the system.

If a direct i/o StoreData or FetchData RPC failed such that it must be retried, the retried RPC would fail due to an attempt to Fetch or Store the wrong amount of data. This is fixed.

Servers are no longer marked down if RPCs fail with RX_CALL_PEER_RESET, RX_CALL_EXCEEDS_WINDOW, or RX_PROTOCOL_ERROR. RPCs that are safe to retry are retried.

Fixed a race between a call entering error state and call completion that can result in a the call remaining in the DALLY state and the connection channel remaining in use. If this occurs during process or system shutdown it can result in a deadlock.

During shutdown cancel any pending delayed aborts to prevent a potential deadlock. If a deadlock occurs when unloading a kernel module a reboot will be required.

Updated cellservdb.conf

New to v0.197.1 (31 August 2020) and v0.198 (10 October 2020)

Prevent Dead vnode has core/unlinkedel/flock panic introduced in v0.197.

New to v0.197 (26 August 2020)

A new callback management framework for UNIX cache managers reduces the expense of processing volume callback RPCs from O(number of vcache objects) to O(1). A significant amount of lock contention has been avoided. The new design reduces the risk of the single callback service worker thread blocking. Delays in processing callbacks on a client can adversely impact fileserver performance and other clients in the cell.

Bulk fetch status RPCs are available on macOS for the first time. Bulk fetch status permits optimistic caching of vnode status information without additional round-trips. Individual fetch status RPCs are no longer issued if a bulk status fails to obtain the required status information.

Hardware accelerated crypto is now available for macOS cache managers. AuriStor's proprietary aes256-cts-hmac-sha1-96 and aes256-cts-hmac-sha512-384 implementations leverage Intel processor extensions: AESNI AVX2 AVX SSE41 SSSE3 to achieve the fastest encrypt, decrypt, sign and verify times for RX packets.

This release optimizes the removal of "._" files that are used to store extended attributes by avoiding unnecessary status fetches when the directory entry is going to be removed.

When removing the final directory entry for an in-use vnode, the directory entry must be silly renamed on the fileserver to prevent removal of the backing vnode. The prior implementation risked blindly renaming over an existing silly rename directory entry.

Behavior change! When the vfs performs a lookup on ".", immediately return the current vnode.

if the object is a mount point, do not perform fakestat and attempt to resolve the target volume root vnode.

do not perform any additional access checks on the vnode. If the caller already knows the vnode the access checks were performed earlier. If the access rights have changed, they will be enforced when the vnode is used just as they would have if the lookup of "." was performed within the vfs.

do not perform a fetch status or fetch data rpcs. Again, the same as if the lookup of "." was performed within the vfs.

Volumes mounted at more than one location in the /afs namespace are problematic on more than one operating system that do not expect directories to have more than one parent. It is particularly problematic if a volume is mounted within itself. Starting with this release any attempt to traverse a mountpoint to the volume containing the mountpoint will fail with ENODEV.

When evaluating volume root vnodes, ensure that the vnode's parent is set to the parent directory of the traversed mountpoint and not the mountpoint. Vnodes without a parent can cause spurious ENOENT errors on Mojave and later.

v0.196 was not publicly released.

New to v0.195 (14 May 2020)

This is a CRITICAL update for AuriStorFS macOS clients.

In Sep 2019 AuriStorFS v0.189 was released which provided faster and less CPU intensive writing of (>64GB) large files to /afs. These improvements introduced a hash collision bug in the store data path of the UNIX cache manager which can result in file corruption. If a hash collision occurs between two or more files that are actively being written to via cached I/O (not direct I/O), dirty data can be discarded from the auristorfs cache before it is written to the fileserver creating a file with a range of zeros (a hole) on the fileserver. This hole might not be visible to the application that wrote the data because the lost data was cached by the operating system. This bug has been fixed in v0.195 and it is for this reason that v0.195 has been designated a CRITICAL release for UNIX/Linux clients.

While debugging a Linux SIGBUS issue, it was observed that receipt of an ICMP network error in response to a transmitted packet could result in termination of an unrelated rx call and could mark a server down. If the terminated call is a StoreData RPC, permanent data loss will occur. All Linux clients derived from the IBM AFS code base experience this bug. The v0.195 release prevents this behavior.

This release includes changes that impact all supported UNIX/Linux cache managers. On macOS there is reduced lock contention between kernel threads when the vcache limit has been reached.

The directory name lookup cache (DNLC) implementation was replaced. The new implementation avoids the use of vcache pointers which did not have associated reference counts, and eliminates the invalidation overhead during callback processing. The DNLC now supports arbitrary directory name lengths; the prior implementation only cached entries with names not exceeding 31 characters.

Prevent matching arbitrary cell name prefixes as aliases. For example "/afs/y" should not be an alias for "your-file-system.com". Some shells, for example "zsh", query the filesystem for names as users type. Delays between typed characters result in filesystem lookups. When this occurs in the /afs dynroot directory, this could result in cellname prefix string matches and the dynamic creation of directory entries for those prefixes.

New to v0.194 (2 April 2020)

This is a CRITICAL release for all macOS users. All prior macOS clients whether AuriStorFS or OpenAFS included a bug that could result in data corruption either when reading or writing.

This release also fixes these other issues:

sign and notarize installer plugin "afscell" bundle. The lack of digital signature prevented the installer from prompting for a cellname on some macOS versions.

prevent potential for corruption when caching locally modified directories.

v0.193 was withdrawn due to a newly introduced bug that could result in data corruption.

New to v0.192 (30 January 2020)

The changes improve stability, efficiency, and scalability. Post-0.189 changes exposed race conditions and reference count errors which can lead to a system panic or deadlock. In addition to addressing these deficiencies this release removes bottlenecks that restricted the number of simultaneous vfs operations that could be processed by the AuriStorFS cache manager. The changes in this release have been successfully tested with greater than 400 simultaneous requests sustained for for several days.

New to v0.191 (16 December 2019)

Restore keyed cache manager capability broken in v0.189.

Add kernel module version string to AuriStorFS Preference Pane.

Other kernel module bug fixes.

New to v0.190 (14 November 2019)

Short-circuit busy volume retries after volume or volume location entry is removed.

New to v0.189 (28 October 2019)

Faster "git status" operation on repositories stored in /afs.

Faster and less CPU intensive writing of (>64GB) large files to /afs. Prior to this release writing files larger than 1TB might not complete. With this release store data throughput is consistent regardless of file size. (See "UNIX Cache Manager large file performance improvements" later in this file).

macOS Catalina (8 October 2019)

AuriStorFS v0.188 released for macOS Catalina (10.15)

New to v0.188 (23 June 2019)

Increased clock resolution for timed waits from 1s to 1ns

Added error handling for rx multi rpcs interrupted by signals

New to v0.186 (29 May 2019)

v0.184 moved the /etc/yfs/cmstate.dat file to /var/yfs. With this change afsd would fail to start if /etc/yfs/cmstate.dat exists but contains invalid state information. This is fixed.

v0.184 introduced a potential deadlock during directory processing. This is fixed.

Handle common error table errors obtained outside an afs_Analyze loop. Map VL errors to ENODEV and RX, RXKAD, RXGK errors to ETIMEDOUT

Log all server down and server up events. Transition events from server probes failed to log messages.

RX RPC networking:

If the RPC initiator successfully completes a call without consuming all of the response data fail the call by sending an RX_PROTOCOL_ERROR ABORT to the acceptor and returning a new error, RX_CALL_PREMATURE_END, to the initiator.
Prior to this change failure to consume all of the response data would be silently ignored by the initiator and the acceptor might resend the unconsumed data until any idle timeout expired. The default idle timeout is 60 seconds.

Avoid transmitting ABORT, CHALLENGE, and RESPONSE packets with an uninitialized sequence number. The sequence number is ignored for these packets but set it to zero.

New to v0.184 (26 March 2019)

The initial congestion window has been reduced from 10 Rx packets to 4. Packet reordering and loss has been observed when sending 10 Rx packets via sendmmsg() in a single burst. The lack of udp packet pacing can also increase the likelihood of transmission stalls due to ack clock variation.

The UNIX Cache Manager underwent major revisions to improve the end user experience by revealing more error codes, improving directory cache efficiency, and overall resiliency. The cache manager implementation was redesigned to be more compatible with operating systems such as Linux and macOS that support restartable system calls. With these changes errors such as "Operation not permitted", "No space left on device", "Quota exceeded", and "Interrupted system call" can be reliably reported to applications. Previously such errors might have been converted to "I/O error".

New to v0.180 (9 November 2018)

RX reliability and performance improvements for high latency and/or lossy network paths such as public wide area networks.

A fix for a macOS firewall triggered kernel panic introduced in v0.177.

New to v0.177 (17 October 2018)

A fix to AuriStor's RX implementation bug introduced in v0.176 that interferes with communication with OpenAFS and IBM Location and File Services.

New to v0.176 (3 October 2018)

AuriStor's RX implementation has undergone a major upgrade of its flow control model. Prior implementations were based on TCP Reno Congestion Control as documented in RFC5681; and SACK behavior that was loosely modelled on RFC2018. The new RX state machine implements SACK based loss recovery as documented in RFC6675, with elements of New Reno from RFC5682 on top of TCP-style congestion control elements as documented in RFC5681. The new RX also implements RFC2861 style congestion window validation.

When sending data the RX peer implementing these changes will be more likely to sustain the maximum available throughput while at the same time improving fairness towards competing network data flows. The improved estimation of available pipe capacity permits an increase in the default maximum window size from 60 packets (84.6 KB) to 128 packets (180.5 KB). The larger window size increases the per call theoretical maximum throughput on a 1ms RTT link from 693 mbit/sec to 1478 mbit/sec and on a 30ms RTT link from 23.1 mbit/sec to 49.39 mbit/sec.

Improve shutdown performance by refusing to give up callbacks to known unreachable file servers and apply a shorter timeout period for the rest.

Permit RXAFSCB_WhoAreYou to be successfully executed after an IBM AFS or OpenAFS fileserver unintentionally requests an RX service upgrade from RXAFSCB to RXYFSCB.

RXAFS timestamps are conveyed in unsigned 32-bit integers with a valid range of 1 Jan 1970 (Unix Epoch) through 07 Feb 2106. UNIX kernel timestamps are stored in 32-bit signed integers with a valid range of 13 Dec 1901 through 19 Jan 2038. This discrepency causes RXAFS timestamps within the 2038-2106 range to display as pre-Epoch.

RX Connection lifecycle management was susceptible to a number of race conditions that could result in assertion failures, the lack of a NAT ping connection to each file server, and the potential reuse of RX connections that should have been discarded.

This release includes a redesigned lifecycle that is thread safe, avoids assertions, prevents NAT ping connection loss, and ensures that discarded connections are not reused.

The 0.174 release unintentionally altered the data structure returned to xstat_cm queries. This release restores the correct wire format.

Since v0.171, if a FetchData RPC fails with a VBUSY error and there is only one reachable fileserver hosting the volume, then the VFS request will immediately with an ETIMEDOUT error ("Connection timed out").

v0.176 corrects three bugs that contributed to this failure condition. One was introduced in v0.171, another in 0.162 and the final one dates to IBM AFS 3.5p1.

The intended behavior is that a cache manager, when all volume sites fail an RPC with a VBUSY error, will sleep for up to 15 seconds and then retry the RPC as if the VBUSY error had never been received. If the RPC continues to receive VBUSY errors from all sites after 100 cycles, the request will be failed with EWOULDBLOCK ("Operation would block") and not ETIMEDOUT.

Prefer VOLMISSING and VOLBUSY error states to network error states when generating error codes to return to the VFS layer. This will result in ENODEV ("No such device") errors when all volume sites return VNOVOL or VOFFLINE errors and EWOULDBLOCK ("Operation would block") errors when all volume sites return VBUSY errors. (v0.176)

New to v0.174 (24 September 2018)

macOS Mojave (10.14) support

New to v0.170 (27 April 2018)

Faster processing of cell configuration information by caching service name to port information.

RX call sequence number rollover to permit calls that require the transmission of more than 5.5TB of data.

Command parser Daylight Saving Time bug fix

Fix a bug that prevented immediate access to a mount point created with "fs mkmount" on the same machine.

Fix the setting of "[afsd] sysnames = " during cache manager startup.

New to v0.168 (6 March 2018)

Corrects "fs setacl -negative" processing [CVE-2018-7168]

Improved reliability for keyed cache managers. More persistent key acquisition renewals.

Major refresh to cellservdb.conf contents.

DNS SRV and DNS AFSDB records now take precedence when use_dns = yes

Kerberos realm hinting provided by
kerberos_realm = [REALM]

DNS host names are resolved instead of reliance on hard coded IP addresses
The cache manager now defaults to sparse dynamic root behavior. Only thiscell and those cells that are assigned aliases are included in /afs directory enumeration at startup. Other cells will be dynamically added upon first access.
Several other quality control improvements.

New to v0.167 (7 December 2017)

Addresses a critical remote denial of service vulnerability [CVE-2017-17432]
Alters the volume location information expiration policy to reduce the risk of single points of failures after volume release operations.
'fs setquota' when issued with quota values larger than 2TB will fail against OpenAFS and IBM AFS file servers
Memory management improvements for the memory caches.

New to v0.164 (11 November 2017)

Internal cache manager redesign. No new functionality.

New to v0.160 (21 September 2017)

Support for OSX High Sierra's new Apple File System (APFS). Customers must upgrade to v0.160 or later before upgrading to OSX High Sierra.
Reduced memory requirements for rx listener thread
Avoid triggering a system panic if an AFS local disk cache file is deleted or becomes inaccessible.
Fixes to "fs" command line output

New to v0.159 (7 August 2017)

Improved failover behavior during volume maintenance operations
Corrected a race that could lead the rx listener thread to enter an infinite loop and cease processing incoming packets.

New to v0.157 (12 July 2017)

Bundled with Heimdal 7.4 to address CVE-2017-11103 (Orpheus' Lyre puts Kerberos to sleep!)
"vos" support for volume quotas larger than 2TB.
"fs flushvolume" works
Fixed a bug that can result in a system panic during server capability testing

New to v0.150

AuriStorFS file server detection improvements

New to v0.149

rxkad encryption is enabled by default. Use "fs setcrypt off" to disable encryption when tokens are available.
Fix a bug in atomic operations on Sierra and El Capitan which could adversely impact Rx behavior.

New to v0.128

Extended attribute ._ files are automatically removed when the associated files are unlinked
Throughput improvements when sending data

New to v0.121

OSX Sierra support

New to v0.117

Cache file moved to a persistent location on local disk
AuriStor File System graphics
Improvements in Background token fetch functionality
Fixed a bug introduced in v0.44 that could result in an operating system crash when enumerating AFS directories containing Unicode file names (v0.106)
El Capitan security changes prevented Finder from deleting files and directories. As of v0.106, the AuriStor OSX client implements the required functionality to permit the DesktopHelperService to securely access the AFS cache as the user permitting Finder to delete files and directories.

Features:

Not vulnerable to OPENAFS-SA-2015-007.
Office 2011 can save to /afs.
Office 2016 can now save files to /afs.
OSX Finder and Preview can open executable documents without triggering a "Corrupted File" warning. .AI, .PDF, .TIFF, .JPG, .DOCX, .XLSX, .PPTX, and other structured documents that might contain scripts were impacted.
All file names are now stored to the file server using Unicode UTF-8 Normalization Form C which is compatible with Microsoft Windows.
All file names are converted to Unicode UTF-8 Normalization Form D for processing by OSX applications.

Known issues:

None

macOS Installer (10.13 High Sierra)

Release Notes

Known Issues

If the Kerberos default realm is not configured, a delay of 6m 59s can occur before the AuriStorFS Backgrounder will acquire tokens and display its icon in the macOS menu. This is the result of macOS performing a Bonjour (MDNS) query in an attempt to discover the local realm.
"aklog" will fail to acquire tokens if the current working directory is located in an inaccessible AFS volume. When the "cwd" is inaccessible, the Heimdal framework is unable to initialize the Kerberos v5 context and reports an out of memory error.

New v2021.05-64 (22 Aug 2025)

The v2021.05-64 release fixes a race condition which can lead to a kernel panic when executing the "unlog" command.

v2021.05-63 (31 May 2025)

When AuriStorFS cache managers added support for per-file ACLs in v0.1, the masking of access rights based upon the inode's S_IRWXU mode bits were skipped when reading objects from AuriStorFS fileservers. The masking continued to be applied when reading objects from OpenAFS fileservers. This release applies S_IRWXU mode bits masking of granted access rights to both AuriStorFS and OpenAFS fileservers.
The existing fake stat mountpoint evaluation logic assumes that the fileserver returned a AFSFetchStatus.ParentVnode that refers to the parent directory from which the mountpoint was located. However, this is only guaranteed to be true if the mountpoint does not have a per-file ACL (which is permitted) and has never been cross-directory hard linked (which is also permitted).
This release improves the error handling logic to fail the fake stat evaluation if the AFSFetchStatus.ParentVnode does not match expectations.
This release fixes a metadata inconsistency when a mountpoint is created when disconnected mode is enabled. A mountpoint should be reported to the vfs as a directory and not a symlink.
The cache manager must track for each inode whether the object's status infomration was received from an AFS3 fileserver or an AuriStorFS fileserver. This release alters when that information is recorded to ensure that it is recorded each time the inode's status information is updated by a fileserver response.
This knowledge is required to properly determine how to interpret the status information.
Directory entries referring to mountpoints and volume roots must not be stored within the Directory Name Lookup Cache (DNLC). Instead, full lookups must be performed each time to ensure proper mountpoint resolution.
Prior to this release locally created mountpoints were stored into the DNLC by mistake.

v2021.05-62 (24 April 2025)

macOS Sequoia v15.4 vs Cisco AnyConnect VPN

Beginning with macOS 15.4, use of Cisco AnyConnect VPN breaks DNS SRV, AFSDB and TXT record lookups performed using the cross-platform DNS resolver API. As a result, clients which rely upon DNS-based zero-configuration resolution of Kerberos realms and/or AFS cells database services are unable to access those resources. This release switches the macOS builds to use the Apple DNS API.
The lookup failures impacted all command line tools, clients and services.
Note: Communication failures can occur if the VPN is configured by the organization to block the transmission of fragmented UDP packets. Set [afsd] rxmaxmtu = 1328 to ensure all RX packets are small enough to traverse the VPN link.
This release improves consistency of vnode metadata interpretation when the status information is received via an RPC which does not return a callback.

v2021.05-61 (31 March 2025)

Instead of allocating the same sized hash table for the vcache and dcache objects regardless of how many elements are expected to be stored, scale the hash table sizes. The vcache hash table will scale from 4096 buckets to 131072 buckets with a target of 4 entries per bucket. The dcache hash tables will scale from 4096 buckets to 65536 buckets with a target of 2 entries per bucket.
Historically, afs cache managers which create symlinks (including mountpoints) locally allocate a fake infinite lifetime callback to permit applications to use the symlink (or mountpoint) without requiring FetchStatus RPCs to be issued to the fileserver. This approach has drawbacks because it means that fileservers will not issue a callback break when the volume containing the symlink is taken offline, moved, deleted, etc. It also means that a callback will not be sent if another cache manager removes the symlink.
Beginning with this release the AuriStorFS cache manager will no longer allocate a fake infinite lifetime callback and will instead rely upon fileserver allocated callbacks.
"fs makemount" is supposed to create a mountpoint even if the target volume does not exist or is inaccessible. v2021.05 introduced support for creating mountpoints which kafs which broke the ability to create mountpoints to non-existent volumes.

v2021.05-60 (6 March 2025)

The cache manager retries fileserver RPCs up to 100 times when the fileserver replies with VBUSY, VOFFLINE or VSALVAGE. The cache manager fails the application request after the limit is reached. Prior to this release, the failure would result in the cache manager returning an ENODEV error to the application. But ENODEV indicates that the required volume does not exist and the fileserver has not indicated that the volume does not exist. The cache manager will now return EWOULDBLOCK to the application because the volume might be temporarily inaccessible.
Many OpenAFS and AuriStorFS fileservers can return VNOVOL as a side effect of a race between a volume going offline so that control can be given to the Volume Management Service (volserver) and an incoming RPC requesting that the volume be attached. A VNOVOL can also be returned because the volume was temporarily deleted from the vice partition prior to being replaced by a new clone, or prior to the fileserver marking the volume group as having been moved to another fileserver.
The fileserver replying to an RPC with VNOVOL cannot be trusted to mean that the volume has been permanently removed unless it is matched by the Location Service reporting the Volume Location (VL) entry has been deleted. From now on when a VNOVOL response is received and the VL entry exists, the cache manager will return EWOULDBLOCK instead of ENODEV to the application.
Prior to this release, any VL error code returned from the Location Service would be interpreted as meaning that the requested VL entry does not exist. Starting with this release only the VL_NOENT error will be interpreted as the VL entry does not exist.
Restrict the internal use of the ENOENT error code so the only use of ENOENT is to indicate that a directory entry does not exist. Previously, ENOENT could be returned if a disk cache entry could not be allocated or found. A disk cache related ENOENT returned when processing a directory might have been confused for the requested directory entry not existing.
Restrict the internal use of the ENODEV error code so that the only meaning of ENODEV is that the required volume does not exist. Previously, ENODEV could be returned due to transient errors when failing to contact the Location Service.

v2021.05-59 (13 February 2025)

The fix for garbage collection of idle unixuser objects introduced in v2021.05-56 was incomplete. As a result, Rx client connections might not be garbage collected.

v2021.05-57 (9 February 2025)

No new changes for macOS.

v2021.05-56 (7 February 2025)

Minor bug fixes to the Rx RPC implementation
The fix for garbage collection of idle unixuser objects introduced in v2021.05-55 was incomplete. As a result, Rx client connections might not be garbage collected.

v2021.05-55 (23 January 2025)

Minor bug fixes to the Rx RPC implementation
During garbage collection of idle unixuser objects, any Rx client connections associated with the unixuser (uid or pag) that were in use by another thread would not have their cache manager reference dropped. As a result the Rx client connections could never be garbage collected which is problematic on systems which churn through many unixusers. The behavior has been changed so that unixusers objects with in use Rx connections will not be freed. This permits future attempts to garbage collect unixusers to do so only after the Rx connections can also be freed.
Although this misbehavior has been present for decades, it is only since v2021.05-52 that the problem has been observed. It is possible that the changes to reduce the holding of the GLOCK has created a window of opportunity for connections to be simultaneously in use during unixuser garbage collection.

v2021.05-54 (19 January 2025)

The UNIX cache manager changes continue AuriStor's efforts to address a report that was received at the end of September. The end user site uses an OpenAFS cell to distribute software for execution by IBM Spectrum LSF (load sharing facility) managed compute nodes. Each of the approximately one dozen compute nodes is configured with approximately 100 processor cores and 1TB of ram. Each job scheduled by LSF runs in a newly allocated Process Authentication Group (PAG). Some jobs run for hours or days; others complete in a few seconds. LSF has been observed scheduling more than 3000 jobs in under a minute with peak counts of cache manager user objects at 1300. Everything ran well until it didn't. After some unexplained trigger event, all of the remaining free memory on the system would be consumed in about an hour causing the system to panic.

Over the last five months AuriStor has diligently worked to develop theories, optimize the AuriStorFS UNIX cache manager to address each theory; then move on to subsequent ideas once prior theories were disproved. Over this time we have replaced the memory management lifecycle for access rights entries and rx events. We have made it possible for Linux permission checks to be processed without acquiring the GLOCK in the majority of access patterns thereby increasing parallelism. The lifecycle management of PAG allocation and user entity tracking has been rewritten to avoid the GLOCK. The Linux inode lifecycle management has been rewritten which some sites attribute to elimination of swapping on their systems. All of these changes increased throughput and parallelism within and without the cache manager. However, none of them prevented the memory consumption once the unknown trigger event occurred.

It is with great pleasure that we can report that the trigger event and the source of the memory consumption have been identified. The trigger is a damaged Rx packet header which causes the AuriStor Rx stack to place an Rx connection into an RX_PROTOCOL_ERROR error state when there is an in-flight call to an OpenAFS fileserver. The memory consumption is due to the infinite loop that occurs when an Rx connection is in an error state and the error code is not one of the error codes which forces replacement of the Rx connection. Within each pass of the loop a structure is allocated to track the history of per-server errors so they can be logged when hard mount mode is enabled for the cell. AuriStor Rx permits up to 12 calls on each Rx connection. As a result there can be up to a dozen processes each spinning and allocating memory when the Rx connection is placed into the error state.

Days before Christmas 2023 AuriStor identified a race condition which can result in damaged Rx DATA packets being written to the network. Initial transmission of Rx DATA packets can be sent either from an application thread or an Rx listener. Packet retransmission occurs either in an Rx listener thread or in the Rx event thread. The race permits a second thread to modify the Rx packet headers while the sendmsg() syscall from the original transmission is executing. This race is possible because the original thread might exhaust its timeslice during the sendmsg() call and it might be many milliseconds before it is rescheduled on a busy system. In the meantime, packet retransmissions might be initiated either by an Rx listener thread processing an ACK packet or the Rx event thread processing a timeout event. If the initial packet transmission completes while the Rx headers are being rewritten the resulting packet sent on the wire might be garbage. AuriStor observed that if the modified packets were jumbograms that silent data corruption could occur so we disabled all use of jumbograms in v2021.05-35 We also notified both IBM and OpenAFS of the problem.

v2021.05-53 modified the RPC result analyze logic to force the replacement of Rx connections in an error state except when the error signals a token expiration. When a token is expired there is no benefit to replacing the connection until new tokens have been acquired. This release further ensures that infinite memory allocation cannot occur by tracking only the most recent error per fileserver; and only does so if hard mount mode is enabled for the cell.

Processes that issued syscalls which became stuck in a RPC retry loop could not be interrupted. In addition to the unintentional retry loops there are intentional retry loops such as those that occur because a fileserver repeated fails the call with a “VBUSY” error. “VBUSY” originally meant that the required volume was temporarily in use by the volume management service (“volserver”) and would be back in service soon so please wait. Beginning with IBM AFS 3.4 the error code was used to force a retry for reasons having nothing to do with the requested volume’s availability. IBM AFS 3.4 introduced the concept of multi-homed clients which meant that the fileserver could no longer track the identity of a cache manager by its IPv4 address. Instead, each cache manager randomly generates a UUID which the fileserver must query each time an incoming RPC is received on a previously unrecognized Rx connection. An Rx connection might be unrecognized because it’s new, or because the client’s IP address changed, or because the fileserver garbage collected its state information. Regardless of the reason, if a call is received and there is no Rx connection binding to a cache manager UUID, then the fileserver application worker thread to which the call was assigned must issue an outgoing RPC to the client’s RXAFSCB service. Each client has at most one worker thread. Prior to this release, every incoming RXAFSCB call required acquisition of the GLOCK before it could execute. If the GLOCK is unavailable, the RXAFSCB thread blocks and no other pending RXAFSCB calls can be processed. If a RXAFSCB call blocks for too long, the fileserver timeouts trigger and the fileserver replies to the incoming RXAFS call with “VBUSY”. “VBUSY” can also be sent if the fileserver’s quota on the number of calls from the same endpoint is exceeded.

The v2021.05-54 release removes the GLOCK requirement for many RXAFSCB RPCs including all of the UUID query RPCs and all whole volume callback RPCs. Callback RPCs for individual vnodes must still acquire the GLOCK. However, it is hoped that these changes will significantly reduce the time required to process the queries generated due to unrecognized Rx connections. The AuriStor Rx stack permits a dozen calls to be issued on a single logical connection which the fileserver sees as three unique Rx connections from the same endpoint.

Each Unix UID or PAG ID must use an independent Rx connection to each fileserver. On a system which processes /afs syscalls for 1200 UID or PAG IDs in a minute, there is the potential for 3600 Rx connections per fileserver. When the fileserver is OpenAFS, this quickly triggers per endpoint throttling and generates VBUSY storms which can result in unlucky client processes experiencing starvation. With this release, starving processes can be interrupted and RXAFSCB UUID queries can be completed without blocking.

AuriStor believes this release will provide significant benefits for large scale compute nodes, web servers and time sharing systems.

v2021.05-53 (4 January 2025)

v2021.05-52 switched the memory allocation strategy for rx_event objects from a global free list to reliance upon the operating system's memory allocator to allocate individual objects which are freed upon rx_event completion or cancelation. Reliance upon the system allocator introduces the risk of allocation failures due to out of memory conditions. If allocation of an rx_event object fails, a reference count leak on the associated rx_connection or rx_call object can occur preventing garbage collection of the object.
This release handles rx_event allocation failures:
- delayed ACK or delayed ABORT events are replaced by immediate transmission of the ACK or ABORT
- other cases fail the call or connection with an RX_OUT_OF_MEMORY error which is translated on the wire to RX_WIRE_CALL_DEAD (-1).
After executing a call to a fileserver, if the Rx connection is in an error state for any reason other than expired tokens, then schedule the Rx connection to be replaced the next time a call is attempted. All calls issued on an Rx connection that is already in an error state fail immediately preventing further communication with the associated fileserver from the UID/PAG.
Rx connections in an expired token error state are not replaced until new tokens are set.
Rx connections can be placed into an error state either due to local or remote failures. Such failures are often generated by the in-use security class (RXKAD/RXGK) but might be due to an out-of-memory condition, system clock advancement or reversal, or even application errors.
This release includes internal code changes to the management of servers, connections and access rights which will be used by a future release to separate PAG ID allocation from the system's UID namespace.
If 'afsd' is configured with a keytab, receipt of a PoweredOn event will force an attempt to acquire system tokens.
On supported macOS versions, enforce the use of code signing identities to authenticate mach messages involving the AFSBackgrounder and the AFSPreference panel.

v2021.05-52 (5 December 2024)

The v2021.05-50 replacement of the linked list implementation of the vcache access cache with a btree and last lookup cache introduced a memory corruption bug which is fixed in this release.
Eliminate the GLOCK requirement for the collection of function call and RPC statistics collection.
Replace small space allocation with a GLOCK free allocator.
Reimplement the cache manager tracing logging routines so they no longer require the GLOCK.
The Rx networking stack schedules many different types of time sensitive events associated with individual packet retransmission, delayed ack and abort processing, path mtu detection, keep alives, and other categories. Prior to this release, all memory allocated to Rx events objects would not be freed to the operating system until the userspace process terminated or the kernel module was unloaded.
Beginning with this release, instead of having a single, global, free list for rxevents, the system allocator is trusted to be able to allocate memory quickly. This simplifies the Rx implementation, and should be more efficient, as most system allocators will have per thread or core slabs, removing the contention of the global lock protecting the rxevent free list. In addition, the change protects against a network event allocating memory that cannot be readily freed.
On non-Total Store Ordering (TSO)architectures, relaxed atomics used as reference counters do not ensure completion of all prior stores or recency of later loads. This release replaces all read-modify-write atomics to be acquire-release, which, among other things, ensures that only the intended ordering is possible.

v2021.05-51 (26 November 2024)

Fix a vcache (aka inode) reference leak that can occur during an access rights query for a non-directory object if the neither the calling process nor the cache manager possesses tokens for the cell containing the object. The leak was introduced in v2021.05-45.
When scanning the token list for each user searching for expired tokens the user spinlock must be held to prevent races. The updated user management implementation pulled into v2021.05-45 failed to hold the spinlock.
During cache manager module shutdown reset global structures to the module initialization state. Doing so improves the likelihood that the module can be reinitialized.

v2021.05-50 (23 November 2024)

The linked list implementation of the vcache access rights cache was replaced with a btree and a last lookup cache. The btree design is more efficient for lookups and removals, and scales better as the number of uids and/or process authentication groups (pags) accessing /afs increases.
Prune access rights for expired users. Doing so reduces the cost of subsequent lookups.
Cache the access rights obtained for a uid|pag via the "anyAccess" field. Previously, a uid|pag that obtained its rights from the "anyAccess" field had to perform and fail a rights lookup on each and every vcache access.
When setting or clearing tokens for a single cell, no longer invalidate the access rights cached for the uid|pag for all cells. Instead, only invalidate the access rights cached for the cell whose tokens were set or cleared.

v2021.05-49 (16 November 2024)

The output of "tokens" command failed to report yfs-rxgk tokens was broken starting in v2021.05-46

v2021.05-48 (12 November 2024)

Preallocated buffer overflows in XDR responses (CVE-2024-10397)
The AuriStorFS and AFS3 RPC suites rely upon Sun RPC XDR to marshal binary data structures for network transfer. The AuriStor XDR implementation is derived from Sun Microsystems' Sun RPC code base. The Sun RPC XDR API permits memory for output parameters to (optionally) be preallocated which can result in various classes of memory corruption and/or memory leaks in RPC initiator processes.
The AuriStorFS v2021.05-48 release introduces additional data length validation checks within the AuriStor XDR implementation and prohibits the use of preallocated memory for string output parameters or fields. All cache managers, servers and command line tools are modified by these changes.

v2021.05-46 (28 October 2024)

Cache Manager:
- Prevent a kernel memory leak when server preferences are set via the yfs-client.conf [afsd] configuration or via "fs setserverprefs".
- Directory enumeration of a truncated directory now returns an error instead of assuming the end of the directory has been reached.
- Since AFS 3.0, the Unix cache manager has used the root identity credentials to create anonymous outgoing connections to the location service and each fileserver. However, if uid 0 is assigned a token, then those Rx connections will no longer be anonymous. Beginning with this release anonymous outgoing connections are always created with the NOPAG identity (uid 0xffffffff) instead of the root identity.
- When establishing an outgoing rxgk connection, do not fallback to the systemuser's credentials if the user's credentials resulted in a fatal error. Falling back to the systemuser's credentials can result in inappropriate use of an anonymous connection.
- Improved access rights cache correctness for YFS servers
  In prior releases, the access check logic used the file rights for any files fetched from an AuriStorFS fileserver. For files fetched from an AFS-3 fileserver (and, historically, for all files), it used the directory rights, with the (a)dmin right from the file mixed in. The (a)dmin right on a non-directory indicates that the object is owned by the authenticated user.
  This approach has some issues when combined with the access rights cache, and current fileserver callback behavior. On an AuriStorFS file server, the rights on a non-directory may be determined by the rights granted on its parent directory or, with per-file ACLs, those granted on the object itself. The fileserver will only break a non-directory's callback when a per-file ACL is changed - changing a directory ACL will not break callbacks on files within that directory. This means that changing a directory ACL will not invalidate access rights cache entries on files in tha directory, even if the effective ACL on this files has changed, and the cached rights are no longer correct.
  This release works around this by adding a new function which returns the access rights for a file hosted on an AuriStor fileserver. It uses the parent vnode information to locate the parent directory. If the parent directory isn't in the cache, or it doesn't have a valid callback, or if it has been changed since the file's access rights were cached, it clears the current access rights. Files without a parent directory must have per-file ACLs, and so their cached rights can be safely used.
  Note that files with parent vnodes may still have per-file ACLs, and that the breadcrumbing performed by the client may add parent vnode fields to vnodes which don't have them provided by the fileserver. Such vnodes may have their cached access rights cleared more frequently than necessary.
- Add a new mechanism for caching access rights within the vcache structure. This cache is protected via a vcache specific spinlock, and can be accessed without holding the GLOCK.
  This new cache mechanism returns the memory associated with cached rights back to the kernel's slab free memory pool instead of adding the unused rights structures to a cache manager managed free list. The previous cache implementation never returned allocated memory to the kernel. Instead, invalidated access rights were appended to a free access rights queue for later reuse.
- When a volume is accessed via multiple mountpoints a choice must be made regarding which mountpoint is considered to be the active (or parent) mountpoint. This release alters the behavior such that the active mountpoint is set every time a mountpoint is traversed.
  This behavior is easier to understand and is more likely to provide the expected result for a single process that repeatedly accesses volumes from multiple mountpoints. However, it can result in unexpected results when multiple processes are traversing multiple mountpoints in parallel without any synchronization.

v2021.05-45 (Not released)

v2021.05-44a (18 September 2024)

Authentication:
- AuriStorFS v2021.05-44 included an updated version of the Heimdal Kerberos framework used by AuriStorFS when acquiring yfs-rxgk and rxkad authentication tokens. The updated Heimdal included a bug which disabled the use of DNS SRV records for KDC discovery and DNS TXT records for realm discovery. As a side effect token acquisition might fail with an unable to reach any KDC in realm error. This is fixed in v2021.05-44a.

v2021.05-44 (17 August 2024)

Cache Manager:
- Since v0.192 the cache manager has failed acquire the global lock when upgrading a shared-lock to a write-lock during the execution of a background cache chunk file truncation.
Authentication:
- Neither MIT nor Heimdal gssapi nor their gss mechanisms consistently initialize the output 'minorStatus' parameter. Various functions can return either success or failure majorStatus values with minorStatus unassigned. As a result, stack garbage will be used when generating error messages. From now on libyfs_acquire will always initialize the minorStatus output variable to zero before calling into the gssapi library.
Command Parser:
- No longer accept the token "-" as a switch which eventually fails with a CMD_UNKNOWNSWITCH error. Instead, process the token as a data value.
- Optimize the processing of the loop which processes "source" command input.
- If the source command input file is "-", read from stdin.

v2021.05-41 (26 June 2024)

Rx Networking (libyfs_rx):
- A race during event creation can lead to the freeing of the event while its still in use.
- RFC1122 says that Net and Host unreachable ICMP errors might be transient and should therefore not be treated as fatal. There is no such language for the equivalent ICMPV6 errors. However, in practice ICMP6_DST_UNREACH_NOROUTE, ICMP6_DST_UNREACH_BEYONDSCOPE, and ICMP6_DST_UNREACH_ADDR can be transient.
  Linux has considered these ICMPV6 destination unreachable errors as non-fatal going back at least as far as the initial git repository commit.
  AuriStor Rx has always treated these as fatal errors which results in immediate termination of in-flight calls when received. Even if the network route corrects itself before the call timeout period expires. This release mirrors the Linux behavior and makes these errors non-fatal.
Cache Manager:
- For the first time the cache manager can detect the deletion of a volume and handle the creation of a new volume with the same name but a different volume id.
- If the location service reports the deletion of a volume, invalidate all mount points to that volume.
- RXAFS_GetCapabilities RPC failures should not be treated as a fatal error preventing failover to another replica site.
Authentication ("libyfs_acquire") used by aklog, vos, pts, bos, afsio:
- rxkad_k5 token acquisition krb5 ccache management. This release altered the krb5 credential cache management strategy once again to work around different bugs in MIT krb5 and Heimdal.
- New ACQUIRE_ERR_CRED_EXPIRED error code introduced to represent the case when a request for a service credential returns one that is already expired.
Command parser (libyfs_cmd):
- When parsing configuration files there is a depth limit of ten active inclusions. This limit was improperly enforced as a limit of ten included files instead of a depth of ten included files. As of this release it is now possible to populate an includedir directory with any number of .conf files.

v2021.05-40

Not released.

v2021.05-39 (20 May 2024)

Parallel Random Number Generation:
- AuriStorFS processes rely upon the krb5_generate_random() and RAND_bytes() functions to obtain random bytes for cryptographic operations and random counters. krb5_generate_random() internally acquires a mutex to protect internal state information. This mutex has become a significant barrier to the encryption and checksumming of Rx packets with both yfs-rxgk and rxkad.
  
  This release replaces general use of krb5_generate_random() and RAND_bytes() with a per-thread ChaCha20 CS-PRNG. This avoids the acquisition of a global mutex and permits increased parallelism on multi-core systems.
Rx Networking (libyfs_rx):
- The Rx network stack schedules a garbage collection operation to execute once per minute. This operation enforces call timeouts, destroys idle connections and destroys idle peers. The operation has historically been performed by the Rx event thread which is already responsible for performing actions in response to call RTOs, sending NAT Ping and keep-alive packets, and retrying connection challenge and reachability checks.
  
  The time complexity of the garbage collection operation is determined by the number of calls, connections, and peers. The busier the Rx endpoint the more work must be performed during each garbage collection run and the longer it takes to complete. While garbage collection is active other events cannot be processed which can interfere with the proper flow control of active calls.
  
  As with all Rx events, the garbage collection event is scheduled to execute at an absolute clock time. If the system clock drifts (or is administratively set) backwards garbage collection will not be performed until the clock catches up with the scheduled time.
  
  Another responsibility of the garbage collection procedure is to terminate calls if the system clock drifted backwards by five minutes or longer. However, when the clocked drifts backwards garbage collection is not performed until the clock has advanced beyond the point where calls require termination. As a result, calls are not terminated due to backwards clock drift and they can stall.
  
  This release re-implements the garbage collection procedure using a dedicated thread and relative waits. This change ensures that the garbage collection procedure will not prevent the execution of call related events and permits calls to be terminated when large backward clock drifts are detected.
Disk Cache Management:
- Since IBM AFS 3.5, the cache has been considered "too full" even if there exist cache files have been discarded but not yet truncated. When the cache is "too full" most operations that write to the cache will block until truncation of discarded cache files has been performed which results in unnecessary delays. This release fixes the cache such that that discarded but not yet truncated cache files do not block write operations.
- This release permits the cache truncation daemon thread to exit sooner if the cache manager is shutting down.
Improved failover when the RXGK service (co-located with each vlserver) fails to issue tokens. The failures might be the result of misconfiguration, an inability to read keys or loss of Ubik quorum.

v2021.05-38 (29 February 2024)

As with other AuriStorFS releases since the beginning of 2024, this release includes additional improvements to the Rx RPC implementation which are related to the possibility of silent data corruption when Rx jumbograms are in use. Prior releases disabled the negotiation of Rx jumbograms such that the v2021.05-37 Rx peer will refuse to send Rx jumbograms and will request that the remote peer does not send them. However, a bad actor could choose to send Rx jumbograms even though they were asked not to. v2021.05-38 introduces additional protections to ensure that a corrupt Rx jumbogram is dropped instead of being accepted.

The v2021.05-38 Rx RPC implementation also includes two optimizations. First, when Rx initiators complete a call they will no longer send an extra ACK packet to the Rx acceptor of the completed call. The sending of this unnecessary ACK creates additional work for the server which can result in increased latency for other calls being processed by the server.

Second, all AuriStor Rx services require a reach check for incoming calls from Rx peers to help protect against Distributed Reflection Denial of Service (DRDoS) attacks and execution of RPCs when the response cannot be delivered to the caller. A new reach check is required for each new call that arrives more than 60 seconds after the prior reach check completed. v2021.05-38 Rx considers the successful acknowledgment of a response DATA packet as a reach check validation. With this change reach checks will not be periodically required for a peer that completes at least one call per 60 seconds. A 1 RTT delay is therefore avoided each time a reach check can be avoided. In addition, reach checks require the service to process an additional ACK packet. Eliminating a large number of reach checks can improve overall service performance.

The final Rx RPC change in this release is specific to kernel implementations. Prior releases restricted the frequency of executing time scheduled Rx events to a granularity no smaller than 500ms. As a result an RTO timer event for a lost packet could not be shorter than 500ms even if the measured RTT for the connection is significantly smaller. The minimum RTO for a connection in AuriStor Rx is 200ms. The inability to schedule shorter timeouts impacts recovery from packet loss.

v2021.05-37 (5 February 2024)

Rx improvements:
- The v2021.05-36 release permanently disabled all use of Rx jumbograms due to a risk of silent data corruption. However, when advertising the number of acceptable datagrams in the ACK trailer a missing htonl() set the value to 16777216 instead of 1 on little-endian systems.
- When sending a PING ACK as a reachability test, ensure that the previousPacket field is properly assigned to the largest accepted DATA packet sequence number instead of zero.
- Replace the initialization state flag with two flags. One that indicates that Rx initialization began and the other that it succeeded. The first prevents multiple attempts at initialization after failure. The second prevents shutdown from accessing uninitialized structures if initialization failed.
Cache Manager Improvements:
- No longer refuse to start if both the 'cachedir' and 'memcache' options are present in the configuration file.
- New variable to store the maximum number of cache blocks used. which is accessible via /proc/fs/auristorfs/cache/blocks_used_max.

v2021.05-36 (10 January 2024)

Rx improvements:
- Permanently disable all use of Rx jumbograms due to a risk of silent data corruption.
  
  Ever since OpenAFS 1.0, and possibly before, a race condition has existed when Rx transmits packets. As the rx_call.lock is dropped when starting packet transmission, there is no protection for data that is being copied into the kernel by sendmsg(). It is critical that this packet data is not modified by another thread. However, races exist between the application, listener, and event threads that can lead to retransmissions starting whilst an original transmission is still in progress. This can lead to the packet headers being overwritten, and either the original transmission, the retransmission or both sending corrupt data to the peer.
  
  This corruption can affect the packet serial number or packet flags. It is particularly harmful when the packet flags are corrupted, as this can lead to multiple Rx packets which were intended to be sent as Rx jumbograms being delivered and misinterpreted as a single large packet. The eventual result of this depends on the Rx security class in play, but it can cause decrypt integrity errors (rxgk:crypt and rxgk:auth) or corruption of the data stream (rxnull, rxgk:clear or rxkad:auth).
  
  All AuriStorFS servers, OpenAFS 1.6 or later servers, and the Windows cache manager have been shipped with Rx jumbograms disabled by default. The UNIX cache managers however are shipped with jumbograms enabled. There are many AFS cells around the world that continue to deploy OpenAFS 1.4 or earlier fileservers which continue to negotiate the use of Rx jumbograms.
  
  It is worth noting that all AuriStorFS v0.198 and later fileservers and cache managers implement explicit checks that will recognize the corrupted application data stream and prevent corrupted file content from being stored either into an AFS vnode's backing store in the volume's object store or the cache manager's AFS cache. OpenAFS cache managers and fileservers do not have these protections.
  
  With Rx jumbograms disabled the maximum number of Rx packets in a datagram is reduced from 6 to 1; the maximum number of send and receive datagram fragments is reduced from 4 to 1; and the maximum advertised MTU is restricted to 1444 - the maximum rx packet size prior to the introduction of jumbograms in IBM AFS 3.5.
- If the rx call flow state transitions from either the RECOVERY or RESCUE states to the LOSS state as a result of an RTO resend event while writing packets to the network, cease transmission of any new DATA packets if there are packets in the resend queue.
- When the call flow state is LOSS and all packets in the resend queue have been retransmitted and yet the recovery point has not been reached, then permit new DATA packets to be sent in order to maintain a full congestion window.
- Add a safety check to prevent the estimated RTT from underflowing when the actual roundtrip time is smaller than 125us.
- Fix the computation of the padding required for rxgk encrypted packets. This bug results in packets sending 8 bytes fewer per packets than the network permits. This bug accidentally prevented the construction of Rx jumbograms when a call is protected by rxgk:crypt.
- Replace the random number generator with a more security source of random bytes.

v2021.05-33 (27 November 2023)

Rx improvements:
- Not all calls transfer enough data to be able to measure a smoothed round-trip time (SRTT). Calls which are unable to compute a SRTT should not be used to update the peer host RTO value which is used to initialize the RTO for subsequent calls.
  
  Without this change, a single DATA packet call will cause the peer host RTO to be reduced to 0ms. Subsequent calls will start with a RTO value of MAX(0, rxi_minPeerTimeout) where rxi_minPeerTimeout defaults to 200ms. If the actual measured RTO is greater than 200ms, then initial RTO will be too small resulting in premature triggering of the RTO timer and the call flow state entering the loss phase which can significantly hurt performance.
- Initialize the peer host RTO to rxi_minPeerTimeout (which defaults to 200ms) instead of one second. Although RFC6298 recommends the use of one second when no SRTT is available, Rx has long used the rxi_minPeerTimeout value for other purposes which are supposed to be consistent with initial RTO value. It should be noted that Linux TCP uses 200ms instead of one second for this purpose.
- If associating a security class with an Rx connection fails immediately place the Rx connection into an error state. A failure might occur if the security class is unable to access valid key material.
- If an incoming Rx call requires authentication and the security class is unable to successfully generate a challenge, put the incoming Rx connection into an error state and issue an abort to the caller.
- If an incoming Rx call requires authentication and the security class is able to generate a challenge but the challenge cannot be returned to Rx, then treat this as a transient error. Do not acknowledge the incoming DATA packet and do not place the Rx connection into an error state. An attempt to re-issue the challenge will be performed when the DATA packet is retransmitted.
- If an Rx call is terminated due to the expiration of the configured connection dead time, idle dead time, hard dead time, or as a result of clock drift, then send an ABORT to the peer notifying them that the call has been terminated. This is particularly important for terminated outgoing calls. If the peer does not know to terminate the call, then the call channel might be in use when the next outgoing call is issued using the same call channel. If the next incoming call is received by an in-use call channel, the receiver must drop the received DATA packet and return a BUSY packet. The call initiator will need to wait for a retransmission timeout to pass before retransmitting the DATA packet. Receipt of BUSY packets cannot be used to keep a call alive and therefore the requested call is at greater risk of timing out if the network path is congested.
aklog and krb5.log (via libyfs_acquire):
- If the linked Kerberos library implements krb5_cc_cache_match() and libacquire has been told to use an explicit principal name and credential cache, the Kerberos library might return KRB5_CC_NOTFOUND even though the requested credential cache is the correct one to use. This release will not call krb5_cc_cache_match() if the requested credential cache contains the requested principal.
Cell Service Database (cellservdb.conf):
- cellservdb.conf has been synchronized with the 31 Oct 2023 update to the grand.central.org CellServDB file.

v2021.05-32 (9 October 2023)

No significant changes for macOS compared to v2021.05-31

v2021.05-31 (25 September 2023)

New platform:
- macOS 14 Sonoma
macOS 14 Sonoma:

AuriStorFS v2021.05-29 and later installers for macOS 13 Ventura are compatible with macOS 14 Sonoma and do not need to be removed before upgrading to macOS 14 Sonoma. Installation of the macOS 14 Sonoma version of AuriStorFS is recommended.

Cache Manager:
- If an AuriStorFS cache manager is unable to use the yfs-rxgk security class when communicating with an AuriStorFS fileserver, it must assume it is IBM AFS 3.6 or OpenAFS and upgrade it to AuriStorFS if an upgrade probe returns a positive result. Once a fileserver's type is identified as AuriStorFS the type should never be reset; even if communication with the fileserver is lost or the fileserver restarts.
  
  If an AuriStorFS fileserver is replaced by an OpenAFS fileserver on the same endpoint, then the UUID of the OpenAFS must be different. As a result, the OpenAFS fileserver will be observed as distinct from the AuriStorFS fileserver that previously shared the endpoint.
  
  Prior to this release there were circumstances in which the cache manager discarded the fileserver type information and would fail to recognize the fileserver as an AuriStorFS fileserver when yfs-rxgk could not be used. This release prevents the cache manager from resetting the type information if the fileserver is marked down.
- If a fileserver's location service entry is updated with a new uniquifier value (aka version number), this indicates that one of the following might have changed:
  1. the fileserver's capabilities
  2. the fileserver's security policy
  3. the fileserver's knowledge of the cell-wide yfs-rxgk key
  4. the fileserver's endpoints
  Beginning with this release the cache manager will force the establishment of new Rx connections to the fileserver when the uniquifier changes. This ensures that the cache manager will attempt to fetch new per-fileserver yfs-rxgk tokens from the cell's RXGK service, enforce the latest security policy, and not end up in a situation where its existing tokens cannot be used to communicate with the fileserver.
aklog:
- Fix incorrect output when populating the server list for a service fails. The stashed extended error explaining the cause of the failure was not displayed.
- If a cell has neither _afs3-prserver._udp. DNS SRV records nor AFSDB records, the lookup of the cell's protection servers would fail if there is no local cell configuration details. The fallback to use _afs3-vlserver._udp. DNS SRV records did not work. This is corrected in this release.

v2021.05-30 (6 September 2023)

Do not mark a fileserver down in response to a KRB5 error code.
fs cleanacl must not store back to the file server a cleaned acl if it was inherited from a directory. Doing so will create a file acl.
Correct the generation of never expire rxkad_krb5 tokens from Kerberos v5 tickets which must have a start time of Unix epoch and an end time of 0xFFFFFFFF seconds. The incorrectly generated tokens were subject to the maximum lifetime of 30 days.
Correct the generation of the yfs-rxgk RESPONSE packet header which failed to specify the key version generation number used to encrypt the authenticator. If the actual key version is greater than zero, then the authenticator would fail to verify.
Enforce a maximum NAT ping period of 20s to ensure that NAT/PAT/firewall rules due not expire while Rx RPCs are in-flight.

v2021.05-29 (26 June 2023)

Execution of fs commands such as examine, whereis, listquota, fetchacl, cleanacl, storeacl, whoami, lsmount, bypassthreshold and getserverprefs could result in memory leaks by the AuriStorFS kernel extension.

v2021.05-27 (1 May 2023)

Fixes for bugs in vos introduced in v2021.05-26.

v2021.05-26 (17 April 2023)

Fixed a potential kernel memory leak when triggered by fs examine, fs listquota, or fs quota.
Increased logging of VBUSY, VOFFLINE, VSALVAGE, and RX_RESTARTING error responses. A log message is now generated whenever a task begins to wait as a result of one of these error responses from a fileserver. Previously, a message was only logged if the volume location information was expired or discarded.
Several changes to optimize internal volume lookups.
Faster failover to replica sites when a fileserver returns RX_RESTARTING, VNOVOL or VMOVED.
rxdebug regains the ability to report rx call flags and rx_connection flags.
The RXRPC library now terminates calls in the QUEUED state when an ABORT packet is received. This clears the call channel making it available to accept another call and reduces the work load on the worker thread pool.
Fileserver endpoint registration changes no longer result in local invalidation of callbacks from that server.
Receipt of an RXAFSCB_InitCallBackState3 RPC from a fileserver no longer resets the volume site status information for all volumes on all servers.

v2021.05-25 (28 December 2022)

The v2021.05-25 release includes further changes to RXRPC to improve reliability. The changes in this release prevent improper packet size growth. Packet size growth should never occur when a call is attempting to recover from packet loss; and is unsafe when the network path's maximum transmission unit is unknown. Packet size growth with be re-enabled in a future AuriStorFS release that includes Path MTU detection and the Extended SACK functionality.
Improved error text describing the source of invalid values in /etc/yfs/yfs-client.conf or included files and directories.

v2021.05-24 (25 October 2022)

New Platform: macOS 13 (Ventura)

v2021.05-22 (12 September 2022) and v2021.05-21 (6 September 2022)

RX RPC
- If receipt of a DATA packet causes an RX call to enter an error state, do not send the ACK of the DATA packet following the ABORT packet. Only send the ABORT packet.
- AuriStor RX has failed to count and report the number of RX BUSY packets that have been sent. Beginning with this change the sent RX BUSY packet count is once again included in the statistics retrieved via rxdebug server port -rxstats.
- Introduce minimum and maximum bounds checks on the ACK packet trailer fields. If the advertised values are out of bounds for the receiving RX stack, do not abort the call but adjust the values to be consistent with the local RX RPC implementation limits. These changes are necessary to handle broken RX RPC implementations or prevent manipulation by attackers.

New to v2021.05-20 (15 August 2022) and v2021.05-19 (13 August 2022)

RX RPC
- Include the DATA packet serial number in the transmitted reachability check PING ACK. This permits the reachability test ACK to be used for RTT measurement.
- Do not terminate a call due to an idle dead timeout if there is data pending in the receive queue when the timeout period expires. Instead deliver the received data to the application. This change prevents idle dead timeouts on slow lossy network paths.
- Fix assignment of RX DATA, CHALLENGE, and RESPONSE packet serial numbers in macOS (KERNEL). Due to a mistake in the implementation of atomic_add_and_read the wrong serial numbers were assigned to outgoing packets.

New to v2021.05-18 (12 July 2022)

Cache Manager
- Prevent a kernel memory leak of less than 64 bytes for each bulkstat RPC issued to a fileserver. Bulkstat RPCs can be frequently issued and over time this small leak can consume a large amount of kernel memory. Leak introduced in AuriStorFS v0.196.
- The Perl::AFS module directly executes pioctls via the OpenAFS compatibility pioctl interface instead of the AuriStorFS pioctl interface. When Perl::AFS is used to store an access control list (ACL), the deprecated RXAFS_StoreACL RPC would be used in place of the newer RXAFS_StoreACL2 or RXYFS_StoreOpaqueACL2 RPCs. This release alters the behavior of the cache manager to use the newer RPCs if available on the fileserver and fallback to the deprecated RPC. The use of the deprecated RPC was restricted to use of the OpenAFS pioctl interface.
RX RPC
- Handle a race during RX connection pool probes that could have resulted in the wrong RX Service ID being returned for a contacted service. Failure to identify that correct service id can result in a degradation of service.
- The Path MTU detection logic sends padded PING ACK packets and requests a PING_RESPONSE ACK be sent if received. This permits the sender of the PING to probe the maximum transmission unit of the path. Under some circumstances attempts were made to send negative padding which resulted in a failure when sending the PING ACK. As a result, the Path MTU could not be measured. This release prevents the use of negative padding.
Preparation for supporting macOS 13 Ventura when it is released in Fall 2022.
Some shells append a slash to an expanded directory name in response to tab completion. These trailing slashes interfered with "fs lsmount", "fs flushmount" and "fs removeacl" processing. This release includes a change to prevent these commands from breaking when presented a trailing slash.

New to v2021.05-17 (16 May 2022)

Cell Service Database Updates
- Update cern.ch, ics.muni.cz, ifh.de, cs.cmu.edu, qatar.cmu.edu, it.kth.se
- Remove uni-hohenheim.de, rz-uni-jena.de, mathematik.uni-stuttgart.de, stud.mathematik.uni-stuttgart.de, wam.umd.edu
- Add ee.cooper.edu
- Restore ams.cern.ch, md.kth.se, italia
Fix parsing of [afsd] rxwindow configuration which can be used to specified a non-default send/receive RX window size. The current default is 128 packets.
RX Updates
- Add nPacketsReflected

Logging improvements

Cache directory validation errors log messages now include the cache directory path.

Log the active configuration path if "debug" logging is enabled.

More details of rxgk token extraction failures.

New to v2021.05-16 (24 March 2022)

RX - Previous releases re-armed the Retransmission Timeout (RTO) each time a new unacknowledged packet is acknowledged instead of when a new leading edge packet is acknowledged. If leading edge data packet and its retransmission are lost, the call can remain in the "recovery" state where it continues to send new data packets until one of the following is true:
. the maximum window size is reached
. the number of lost and resent packets equals 'cwind'
at which point there is nothing left to transmit. The leading edge data packet can only be retransmitted when entering the "loss" state but since the RTO is reset with each acknowledged packet the call stalls for one RTO period after the last transmitted data packet is acknowledged.

This poor behavior is less noticiable with small window sizes and short lived calls. However, as window sizes and round-trip times increase the impact of a twice lost packet becomes significant.

RX - Never set the high-order bit of the Connection Epoch field. RX peers starting with IBM AFS 3.1b through AuriStor RX v0.191 ignore the source endpoint when matching incoming packets to RX connections if the high-order epoch bit is set. Ignoring the source endpoint is problematic because it can result in a call entering a zombie state whereby all PING ACK packets are immediately responded to the source endpoint of the PING ACK but any delayed ACK or DATA packets are sent to the endpoint bound to the RX connection. An RX client that moves from one network to another or which has a NAT|PAT device between it and the service can find themselves stuck.

Starting with AuriStor RX v0.192 the high-order bit is ignored by AuriStor RX peer when receiving packets. This change to always clear the bit prevents IBM AFS and OpenAFS peers from ignoring the source endpoint.

RX - The initial packetSize calculation for a call is altered to require that all constructed packets before the receipt of the first ACK packet are eligible for use in jumbograms if and only if the local RX stack has jumbograms enabled and the maximum MTU is large enough. By default jumbograms are disabled for all AuriStorFS services. This change will have a beneficial impact if jumbograms are enabled via configuration; or when testing RX performance with "rxperf".

New fs whereis -noresolve option displays the fileservers by network endpoint instead of DNS PTR record hostname.

New to v2021.05-15 (24 January 2022)

kernel - fixed YFS_RXGK service rx connection pool leak

New to v2021.05-14 (20 January 2022)

fs mkmount permit mount point target strings longer than 63 characters.
afsd enhance logging of yfs-rxgk token renewal errors.

afsd gains a "principal = configuration option for use with keytab acquisition of yfs-rxgk tokens for the cache manager identity.

kernel - Avoid unnecessary rx connection replacement by racing threads after token replacement or expiration.

kernel - Fix a regression introduced in v2021.05 where an anonymous combined identity yfs-rxgk token would be replaced after three minutes resulting in the connection switching from yfs-rxgk to rxnull.

kernel - Fix a regression introduced in v0.208 which prevented the invalidation of cached access rights in response to a fileserver callback rpc. The cache would be updated after the first FetchStatus rpc after invalidation.

kernel - Reset combined identity yfs-rxgk tokens when the system token is replaced.

kernel - The replacement of rx connection bundles in the cache manager to permit more than four simultaneous rx calls per uid/pag with trunked rx connections introduced the following regressions in v2021.05.

a memory leak of discarded rx connection objects

failure of NAT ping probes after replacement of an connection

inappropriate use of rx connections after a service upgrade failure

All of these regressions are fixed in patch 14.

New to v2021.05-12 (7 October 2021)

fs ignorelist -type afsmountdir in prior releases could prevent access to /afs.

Location server rpc timeout restored to two minutes instead of twenty minutes.

Location server reachability probe timeout restored to six seconds instead of fifty seconds.

Cell location server upcall results are now cached for fifteen seconds.

Multiple kernel threads waiting for updated cell location server reachability probes now share the results of a single probe.

RX RPC implementation lock hierarchy modified to prevent a lock inversion.

RX RPC client connection reference count leak fixed.

RX RPC deadlock during failed connection service upgrade attempt fixed..

New to v2021.05-9 (25 October 2021)

First public release for macOS 12 Monterey build using XCode 13. When upgrading macOS to Monterey from earlier macOS releases, please upgrade AuriStorFS to v2021.05-9 on the starting macOS release, upgrade to Monterey and then install the Monterey specific v2021.05-9 release.

Improved logging of "afsd" shutdown when "debug" mode is enabled.

Minor RX network stack improvements

New to v2021.05-3 (10 June 2021)

Fix for [cells] cellname = {...} without server list.

New to v2021.05 (31 May 2021)

Multi-homed location servers are finally managed as a single server instead of treating each endpoint as a separate server. The new functionality is a part of the wholesale replacement of the former cell management infrastructure. Location server communication is now entirely managed as a cluster of multi-homed servers for each cell. The new infrastructure does not rely upon the global lock for thread safety.

This release introduces a new infrastructure for managing user/pag entities and tracking their per cell tokens and related connection pools.

Expired tokens are no longer immediately deleted so that its possible for them to be listed by "tokens" for up to two hours.

Prevent a lock inversion introduced in v0.208 that can result in a deadlock involving the GLOCK and the rx call.lock. The deadlock can occur if a cell's list of location servers expires and during the rebuild an rx abort is issued.

Add support for rxkad "auth" mode rx connections in addition to "clear" and "crypt". "auth" mode provides integrity protection without privacy.

Add support for yfs-rxgk "clear" and "auth" rx connection modes.

Do not leak a directory buffer page reference when populating a directory page fails.

Re-initialize state when populating a disk cache entry using the fast path fails and a retry is performed using the slow path. If the data version changes between the attempts it is possible for truncated disk cache data to be treated as valid.

Log warnings if a directory lookup operation fails with an EIO error. An EIO error indicates that an invalid directory header, page header, or directory entry was found.

Do not overwrite RX errors with local errors during Direct-I/O and StoreMini operations. Doing so can result in loss of VBUSY, VOFFLINE, UAENOSPC, and similar errors.

Correct a direct i/o code path which could overwrite a fileserver returned error code with a local RXGEN_CC_UNMARSHAL error.

Correct the StoreMini code path which could overwrite a fileserver returned error code with a local RXGEN_CC_UNMARSHAL error.

Ensure the rx call object is not locked when writing to the network socket.

Removed all knowledge of the KERNEL global lock from RX. Acquiring the GLOCK from RX is never safe if any other lock is held. Doing so is a lock order violation that can result in deadlocks.

Fixed a race in the opr_reservation system that could produce a cache entry reference undercount.

If a directory hash chain contains a circular link, a buffer page reference could be leaked for each traversal.

Each AFS3 directory header and page header contains a magic tag value that can be used in a consistency check but was not previously checked before use of each header. If the header memory is zero filled during a lookup, the search would fail producing an ENOENT error. Starting with this release the magic tag values are validated on each use. An EIO error is returned if there is a tag mismatch.

"fs setcrypt -crypt auth" is now a permitted value. The "auth" mode provides integrity protection but no privacy protection.

Add new "aklog -levels " option which permits requesting "clear" and "auth" modes for use with yfs-rxgk.

Update MKShim to Apple OpenSource MITKerberosShim-79.

Report KLL errors via a notification instead of throwing an exception which (if not caught) will result in process termination.

If an exception occurs while executing "unlog" catch it and ignore it. Otherwise, the process will terminate.

New to v2021.04 (22 April 2021)

Primarily bug fixes for issues that have been present for years.

A possibility of an infinite kernel loop if a rare file write / truncate pattern occurs.

A bug in silly rename handling that can prevent cache manager initiated garbage collection of vnodes.

New to v0.209 (13 March 2021)

fs setserverprefs and fs getserverprefs updated to support IPv6 and CIDR specifications.

Improved error handling during fetch data and store data operations.

Prevents a race between two vfs operations on the same directory which can result in caching of out of date directory contents.

Use cached mount point target information instead of evaluating the mount point's target upon each access.

Avoid rare data cache thrashing condition.

Prevent infinite loop if a disk cache error occurs after the first page in a chunk is written.

Network errors are supposed to be returned to userspace as ETIMEDOUT. Previously some were returned as EIO.

When authentication tokens expire, reissue the fileserver request anonymously. If the anonymous user does not have permission either EACCES or EPERM will be returned as the error to userspace. Previously the vfs request would fail with an RXKADEXPIRED or RXGKEXPIRED error.

If growth of an existing connection vector fails, wait on a call slot in a previously created connection instead of failing the vfs request.

Volume and fileserver location query infrastructure has been replaced with a new modern implementation.

Replace the cache manager's token management infrastructure with a new modern implementation.

New to v0.206 (12 January 2021) - Bug fixes

Prevents a possible panic during unmount of /afs.

Improved failover and retry logic for offline volumes.

New to v0.205 (24 December 2020) - Bug fixes

Volume name-to-id cache improvements

Fix expiration of name-to-id cache entries

Control volume name-to-id via sysctl

Query volume name-to-id statistics via sysctl

Improve error handling for offline volumes

Fix installer to prevent unnecessary installation of Rosetta 2 on Apple Silicon

New to v0.204 (25 November 2020) - Bug fix for macOS Big Sur

v0.204 prevents a kernel panic on Big Sur when AuriStorFS is stopped and restarted without an operating system reboot.

introduces a volume name-to-id cache independent of the volume location cache.

New to v0.203 (13 November 2020) - Bug fix for macOS

v0.203 prevents a potential kernel panic due to network error.

New to v0.201 (12 November 2020) - Universal Big Sur (11.0) release for Apple Silicon and Intel

v0.201 introduces a new cache manager architecture on all macOS versions except for High Sierra (10.12). The new architecture includes a redesign of:

kernel extension load

kernel extension unload (not available on Big Sur)

/afs mount

/afs unmount

userspace networking

The conversion to userspace networking will have two user visible impacts for end users:
The Apple Firewall as configured by System Preferences -> Security & Privacy -> Firewall is now enforced. The "Automatically allow downloaded signed software to receive incoming connections" includes AuriStorFS.

Observed network throughput is likely to vary compared to previous releases.

On Catalina the "Legacy Kernel Extension" warnings that were displayed after boot with previous releases of AuriStorFS are no longer presented with v0.201.

AuriStorFS /afs access is expected to continue to function when upgrading from Mojave or Catalina to Big Sur. However, as AuriStorFS is built specifically for each macOS release, it is recommended that end users install a Big Sur specific AuriStorFS package.
AuriStorFS on Apple Silicon supports hardware accelerated aes256-cts-hmac-sha1-96 and aes128-cts-hmac-sha1-96 using AuriStor's proprietary implementation.

New to v0.200 (4 November 2020) - Final release for macOS El Capitan (10.11)

The network path between a client and a server often traverses one or more network segments separated by NAT/PAT devices. If a NAT/PAT times out a RPCs endpoint translation mid-call, this can result in an extended delay before failure and the server being marked down, or worse, a call that never terminates and a client that appears to hang until the fileserver is restarted.

This release includes significant changes to the RX stack and the UNIX cache manager to detect such conditions, fail the calls quickly and detect when it is safe to retry the RPC.

NAT/PAT devices that drop endpoint mappings while in use are anti-social and can result in unwanted delays and even data loss. Whenever possible they should be avoided when they can be. That said, the changes in this release are a huge step to making the loss of endpoint mappings tolerable.

Fix segmentation fault of Backgrounder when krb5_get_credentials() fails due to lack of network connectivity.

Fix the "afsd" rxbind option which was ignored if the default port, 7001, is in use by another process on the system.

If a direct i/o StoreData or FetchData RPC failed such that it must be retried, the retried RPC would fail due to an attempt to Fetch or Store the wrong amount of data. This is fixed.

Servers are no longer marked down if RPCs fail with RX_CALL_PEER_RESET, RX_CALL_EXCEEDS_WINDOW, or RX_PROTOCOL_ERROR. RPCs that are safe to retry are retried.

Fixed a race between a call entering error state and call completion that can result in a the call remaining in the DALLY state and the connection channel remaining in use. If this occurs during process or system shutdown it can result in a deadlock.

During shutdown cancel any pending delayed aborts to prevent a potential deadlock. If a deadlock occurs when unloading a kernel module a reboot will be required.

Updated cellservdb.conf

New to v0.197.1 (31 August 2020) and v0.198 (10 October 2020)

Prevent Dead vnode has core/unlinkedel/flock panic introduced in v0.197.

New to v0.197 (26 August 2020)

A new callback management framework for UNIX cache managers reduces the expense of processing volume callback RPCs from O(number of vcache objects) to O(1). A significant amount of lock contention has been avoided. The new design reduces the risk of the single callback service worker thread blocking. Delays in processing callbacks on a client can adversely impact fileserver performance and other clients in the cell.

Bulk fetch status RPCs are available on macOS for the first time. Bulk fetch status permits optimistic caching of vnode status information without additional round-trips. Individual fetch status RPCs are no longer issued if a bulk status fails to obtain the required status information.

Hardware accelerated crypto is now available for macOS cache managers. AuriStor's proprietary aes256-cts-hmac-sha1-96 and aes256-cts-hmac-sha512-384 implementations leverage Intel processor extensions: AESNI AVX2 AVX SSE41 SSSE3 to achieve the fastest encrypt, decrypt, sign and verify times for RX packets.

This release optimizes the removal of "._" files that are used to store extended attributes by avoiding unnecessary status fetches when the directory entry is going to be removed.

When removing the final directory entry for an in-use vnode, the directory entry must be silly renamed on the fileserver to prevent removal of the backing vnode. The prior implementation risked blindly renaming over an existing silly rename directory entry.

Behavior change! When the vfs performs a lookup on ".", immediately return the current vnode.

if the object is a mount point, do not perform fakestat and attempt to resolve the target volume root vnode.

do not perform any additional access checks on the vnode. If the caller already knows the vnode the access checks were performed earlier. If the access rights have changed, they will be enforced when the vnode is used just as they would have if the lookup of "." was performed within the vfs.

do not perform a fetch status or fetch data rpcs. Again, the same as if the lookup of "." was performed within the vfs.

Volumes mounted at more than one location in the /afs namespace are problematic on more than one operating system that do not expect directories to have more than one parent. It is particularly problematic if a volume is mounted within itself. Starting with this release any attempt to traverse a mountpoint to the volume containing the mountpoint will fail with ENODEV.

When evaluating volume root vnodes, ensure that the vnode's parent is set to the parent directory of the traversed mountpoint and not the mountpoint. Vnodes without a parent can cause spurious ENOENT errors on Mojave and later.

v0.196 was not publicly released.

New to v0.195 (14 May 2020)

This is a CRITICAL update for AuriStorFS macOS clients.

In Sep 2019 AuriStorFS v0.189 was released which provided faster and less CPU intensive writing of (>64GB) large files to /afs. These improvements introduced a hash collision bug in the store data path of the UNIX cache manager which can result in file corruption. If a hash collision occurs between two or more files that are actively being written to via cached I/O (not direct I/O), dirty data can be discarded from the auristorfs cache before it is written to the fileserver creating a file with a range of zeros (a hole) on the fileserver. This hole might not be visible to the application that wrote the data because the lost data was cached by the operating system. This bug has been fixed in v0.195 and it is for this reason that v0.195 has been designated a CRITICAL release for UNIX/Linux clients.

While debugging a Linux SIGBUS issue, it was observed that receipt of an ICMP network error in response to a transmitted packet could result in termination of an unrelated rx call and could mark a server down. If the terminated call is a StoreData RPC, permanent data loss will occur. All Linux clients derived from the IBM AFS code base experience this bug. The v0.195 release prevents this behavior.

This release includes changes that impact all supported UNIX/Linux cache managers. On macOS there is reduced lock contention between kernel threads when the vcache limit has been reached.

The directory name lookup cache (DNLC) implementation was replaced. The new implementation avoids the use of vcache pointers which did not have associated reference counts, and eliminates the invalidation overhead during callback processing. The DNLC now supports arbitrary directory name lengths; the prior implementation only cached entries with names not exceeding 31 characters.

Prevent matching arbitrary cell name prefixes as aliases. For example "/afs/y" should not be an alias for "your-file-system.com". Some shells, for example "zsh", query the filesystem for names as users type. Delays between typed characters result in filesystem lookups. When this occurs in the /afs dynroot directory, this could result in cellname prefix string matches and the dynamic creation of directory entries for those prefixes.

New to v0.194 (2 April 2020)

This is a CRITICAL release for all macOS users. All prior macOS clients whether AuriStorFS or OpenAFS included a bug that could result in data corruption either when reading or writing.

This release also fixes these other issues:

sign and notarize installer plugin "afscell" bundle. The lack of digital signature prevented the installer from prompting for a cellname on some macOS versions.

prevent potential for corruption when caching locally modified directories.

v0.193 was withdrawn due to a newly introduced bug that could result in data corruption.

New to v0.192 (30 January 2020)

The changes improve stability, efficiency, and scalability. Post-0.189 changes exposed race conditions and reference count errors which can lead to a system panic or deadlock. In addition to addressing these deficiencies this release removes bottlenecks that restricted the number of simultaneous vfs operations that could be processed by the AuriStorFS cache manager. The changes in this release have been successfully tested with greater than 400 simultaneous requests sustained for for several days.

New to v0.191 (16 December 2019)

Restore keyed cache manager capability broken in v0.189.

Add kernel module version string to AuriStorFS Preference Pane.

Other kernel module bug fixes.

New to v0.190 (14 November 2019)

Short-circuit busy volume retries after volume or volume location entry is removed.

New to v0.189 (28 October 2019)

Faster "git status" operation on repositories stored in /afs.

Faster and less CPU intensive writing of (>64GB) large files to /afs. Prior to this release writing files larger than 1TB might not complete. With this release store data throughput is consistent regardless of file size. (See "UNIX Cache Manager large file performance improvements" later in this file).

macOS Catalina (8 October 2019)

AuriStorFS v0.188 released for macOS Catalina (10.15)

New to v0.188 (23 June 2019)

Increased clock resolution for timed waits from 1s to 1ns

Added error handling for rx multi rpcs interrupted by signals

New to v0.186 (29 May 2019)

v0.184 moved the /etc/yfs/cmstate.dat file to /var/yfs. With this change afsd would fail to start if /etc/yfs/cmstate.dat exists but contains invalid state information. This is fixed.

v0.184 introduced a potential deadlock during directory processing. This is fixed.

Handle common error table errors obtained outside an afs_Analyze loop. Map VL errors to ENODEV and RX, RXKAD, RXGK errors to ETIMEDOUT

Log all server down and server up events. Transition events from server probes failed to log messages.

RX RPC networking:

If the RPC initiator successfully completes a call without consuming all of the response data fail the call by sending an RX_PROTOCOL_ERROR ABORT to the acceptor and returning a new error, RX_CALL_PREMATURE_END, to the initiator.
Prior to this change failure to consume all of the response data would be silently ignored by the initiator and the acceptor might resend the unconsumed data until any idle timeout expired. The default idle timeout is 60 seconds.

Avoid transmitting ABORT, CHALLENGE, and RESPONSE packets with an uninitialized sequence number. The sequence number is ignored for these packets but set it to zero.

New to v0.184 (26 March 2019)

The initial congestion window has been reduced from 10 Rx packets to 4. Packet reordering and loss has been observed when sending 10 Rx packets via sendmmsg() in a single burst. The lack of udp packet pacing can also increase the likelihood of transmission stalls due to ack clock variation.

The UNIX Cache Manager underwent major revisions to improve the end user experience by revealing more error codes, improving directory cache efficiency, and overall resiliency. The cache manager implementation was redesigned to be more compatible with operating systems such as Linux and macOS that support restartable system calls. With these changes errors such as "Operation not permitted", "No space left on device", "Quota exceeded", and "Interrupted system call" can be reliably reported to applications. Previously such errors might have been converted to "I/O error".

New to v0.180 (9 November 2018)

RX reliability and performance improvements for high latency and/or lossy network paths such as public wide area networks.

A fix for a macOS firewall triggered kernel panic introduced in v0.177.

New to v0.177 (17 October 2018)

A fix to AuriStor's RX implementation bug introduced in v0.176 that interferes with communication with OpenAFS and IBM Location and File Services.

New to v0.176 (3 October 2018)

AuriStor's RX implementation has undergone a major upgrade of its flow control model. Prior implementations were based on TCP Reno Congestion Control as documented in RFC5681; and SACK behavior that was loosely modelled on RFC2018. The new RX state machine implements SACK based loss recovery as documented in RFC6675, with elements of New Reno from RFC5682 on top of TCP-style congestion control elements as documented in RFC5681. The new RX also implements RFC2861 style congestion window validation.

When sending data the RX peer implementing these changes will be more likely to sustain the maximum available throughput while at the same time improving fairness towards competing network data flows. The improved estimation of available pipe capacity permits an increase in the default maximum window size from 60 packets (84.6 KB) to 128 packets (180.5 KB). The larger window size increases the per call theoretical maximum throughput on a 1ms RTT link from 693 mbit/sec to 1478 mbit/sec and on a 30ms RTT link from 23.1 mbit/sec to 49.39 mbit/sec.

Improve shutdown performance by refusing to give up callbacks to known unreachable file servers and apply a shorter timeout period for the rest.

Permit RXAFSCB_WhoAreYou to be successfully executed after an IBM AFS or OpenAFS fileserver unintentionally requests an RX service upgrade from RXAFSCB to RXYFSCB.

RXAFS timestamps are conveyed in unsigned 32-bit integers with a valid range of 1 Jan 1970 (Unix Epoch) through 07 Feb 2106. UNIX kernel timestamps are stored in 32-bit signed integers with a valid range of 13 Dec 1901 through 19 Jan 2038. This discrepency causes RXAFS timestamps within the 2038-2106 range to display as pre-Epoch.

RX Connection lifecycle management was susceptible to a number of race conditions that could result in assertion failures, the lack of a NAT ping connection to each file server, and the potential reuse of RX connections that should have been discarded.

This release includes a redesigned lifecycle that is thread safe, avoids assertions, prevents NAT ping connection loss, and ensures that discarded connections are not reused.

The 0.174 release unintentionally altered the data structure returned to xstat_cm queries. This release restores the correct wire format.

Since v0.171, if a FetchData RPC fails with a VBUSY error and there is only one reachable fileserver hosting the volume, then the VFS request will immediately with an ETIMEDOUT error ("Connection timed out").

v0.176 corrects three bugs that contributed to this failure condition. One was introduced in v0.171, another in 0.162 and the final one dates to IBM AFS 3.5p1.

The intended behavior is that a cache manager, when all volume sites fail an RPC with a VBUSY error, will sleep for up to 15 seconds and then retry the RPC as if the VBUSY error had never been received. If the RPC continues to receive VBUSY errors from all sites after 100 cycles, the request will be failed with EWOULDBLOCK ("Operation would block") and not ETIMEDOUT.

Prefer VOLMISSING and VOLBUSY error states to network error states when generating error codes to return to the VFS layer. This will result in ENODEV ("No such device") errors when all volume sites return VNOVOL or VOFFLINE errors and EWOULDBLOCK ("Operation would block") errors when all volume sites return VBUSY errors. (v0.176)

New to v0.174 (24 September 2018)

macOS Mojave (10.14) support

New to v0.170 (27 April 2018)

Faster processing of cell configuration information by caching service name to port information.

RX call sequence number rollover to permit calls that require the transmission of more than 5.5TB of data.

Command parser Daylight Saving Time bug fix

Fix a bug that prevented immediate access to a mount point created with "fs mkmount" on the same machine.

Fix the setting of "[afsd] sysnames = " during cache manager startup.

New to v0.168 (6 March 2018)

Corrects "fs setacl -negative" processing [CVE-2018-7168]

Improved reliability for keyed cache managers. More persistent key acquisition renewals.

Major refresh to cellservdb.conf contents.

DNS SRV and DNS AFSDB records now take precedence when use_dns = yes

Kerberos realm hinting provided by
kerberos_realm = [REALM]

DNS host names are resolved instead of reliance on hard coded IP addresses
The cache manager now defaults to sparse dynamic root behavior. Only thiscell and those cells that are assigned aliases are included in /afs directory enumeration at startup. Other cells will be dynamically added upon first access.
Several other quality control improvements.

New to v0.167 (7 December 2017)

Addresses a critical remote denial of service vulnerability [CVE-2017-17432]
Alters the volume location information expiration policy to reduce the risk of single points of failures after volume release operations.
'fs setquota' when issued with quota values larger than 2TB will fail against OpenAFS and IBM AFS file servers
Memory management improvements for the memory caches.

New to v0.164 (11 November 2017)

Internal cache manager redesign. No new functionality.

New to v0.160 (21 September 2017)

Support for OSX High Sierra's new Apple File System (APFS). Customers must upgrade to v0.160 or later before upgrading to OSX High Sierra.
Reduced memory requirements for rx listener thread
Avoid triggering a system panic if an AFS local disk cache file is deleted or becomes inaccessible.
Fixes to "fs" command line output

New to v0.159 (7 August 2017)

Improved failover behavior during volume maintenance operations
Corrected a race that could lead the rx listener thread to enter an infinite loop and cease processing incoming packets.

New to v0.157 (12 July 2017)

Bundled with Heimdal 7.4 to address CVE-2017-11103 (Orpheus' Lyre puts Kerberos to sleep!)
"vos" support for volume quotas larger than 2TB.
"fs flushvolume" works
Fixed a bug that can result in a system panic during server capability testing

New to v0.150

AuriStorFS file server detection improvements

New to v0.149

rxkad encryption is enabled by default. Use "fs setcrypt off" to disable encryption when tokens are available.
Fix a bug in atomic operations on Sierra and El Capitan which could adversely impact Rx behavior.

New to v0.128

Extended attribute ._ files are automatically removed when the associated files are unlinked
Throughput improvements when sending data

New to v0.121

OSX Sierra support

New to v0.117

Cache file moved to a persistent location on local disk
AuriStor File System graphics
Improvements in Background token fetch functionality
Fixed a bug introduced in v0.44 that could result in an operating system crash when enumerating AFS directories containing Unicode file names (v0.106)
El Capitan security changes prevented Finder from deleting files and directories. As of v0.106, the AuriStor OSX client implements the required functionality to permit the DesktopHelperService to securely access the AFS cache as the user permitting Finder to delete files and directories.

Features:

Not vulnerable to OPENAFS-SA-2015-007.
Office 2011 can save to /afs.
Office 2016 can now save files to /afs.
OSX Finder and Preview can open executable documents without triggering a "Corrupted File" warning. .AI, .PDF, .TIFF, .JPG, .DOCX, .XLSX, .PPTX, and other structured documents that might contain scripts were impacted.
All file names are now stored to the file server using Unicode UTF-8 Normalization Form C which is compatible with Microsoft Windows.
All file names are converted to Unicode UTF-8 Normalization Form D for processing by OSX applications.

Known issues:

None

macOS Installer (10.12 Sierra)

Release Notes

Known Issues

If the Kerberos default realm is not configured, a delay of 6m 59s can occur before the AuriStorFS Backgrounder will acquire tokens and display its icon in the macOS menu. This is the result of macOS performing a Bonjour (MDNS) query in an attempt to discover the local realm.
"aklog" will fail to acquire tokens if the current working directory is located in an inaccessible AFS volume. When the "cwd" is inaccessible, the Heimdal framework is unable to initialize the Kerberos v5 context and reports an out of memory error.

New v2021.05-64 (22 Aug 2025)

The v2021.05-64 release fixes a race condition which can lead to a kernel panic when executing the "unlog" command.

v2021.05-63 (31 May 2025)

When AuriStorFS cache managers added support for per-file ACLs in v0.1, the masking of access rights based upon the inode's S_IRWXU mode bits were skipped when reading objects from AuriStorFS fileservers. The masking continued to be applied when reading objects from OpenAFS fileservers. This release applies S_IRWXU mode bits masking of granted access rights to both AuriStorFS and OpenAFS fileservers.
The existing fake stat mountpoint evaluation logic assumes that the fileserver returned a AFSFetchStatus.ParentVnode that refers to the parent directory from which the mountpoint was located. However, this is only guaranteed to be true if the mountpoint does not have a per-file ACL (which is permitted) and has never been cross-directory hard linked (which is also permitted).
This release improves the error handling logic to fail the fake stat evaluation if the AFSFetchStatus.ParentVnode does not match expectations.
This release fixes a metadata inconsistency when a mountpoint is created when disconnected mode is enabled. A mountpoint should be reported to the vfs as a directory and not a symlink.
The cache manager must track for each inode whether the object's status infomration was received from an AFS3 fileserver or an AuriStorFS fileserver. This release alters when that information is recorded to ensure that it is recorded each time the inode's status information is updated by a fileserver response.
This knowledge is required to properly determine how to interpret the status information.
Directory entries referring to mountpoints and volume roots must not be stored within the Directory Name Lookup Cache (DNLC). Instead, full lookups must be performed each time to ensure proper mountpoint resolution.
Prior to this release locally created mountpoints were stored into the DNLC by mistake.

v2021.05-62 (24 April 2025)

macOS Sequoia v15.4 vs Cisco AnyConnect VPN

Beginning with macOS 15.4, use of Cisco AnyConnect VPN breaks DNS SRV, AFSDB and TXT record lookups performed using the cross-platform DNS resolver API. As a result, clients which rely upon DNS-based zero-configuration resolution of Kerberos realms and/or AFS cells database services are unable to access those resources. This release switches the macOS builds to use the Apple DNS API.
The lookup failures impacted all command line tools, clients and services.
Note: Communication failures can occur if the VPN is configured by the organization to block the transmission of fragmented UDP packets. Set [afsd] rxmaxmtu = 1328 to ensure all RX packets are small enough to traverse the VPN link.
This release improves consistency of vnode metadata interpretation when the status information is received via an RPC which does not return a callback.

v2021.05-61 (31 March 2025)

Instead of allocating the same sized hash table for the vcache and dcache objects regardless of how many elements are expected to be stored, scale the hash table sizes. The vcache hash table will scale from 4096 buckets to 131072 buckets with a target of 4 entries per bucket. The dcache hash tables will scale from 4096 buckets to 65536 buckets with a target of 2 entries per bucket.
Historically, afs cache managers which create symlinks (including mountpoints) locally allocate a fake infinite lifetime callback to permit applications to use the symlink (or mountpoint) without requiring FetchStatus RPCs to be issued to the fileserver. This approach has drawbacks because it means that fileservers will not issue a callback break when the volume containing the symlink is taken offline, moved, deleted, etc. It also means that a callback will not be sent if another cache manager removes the symlink.
Beginning with this release the AuriStorFS cache manager will no longer allocate a fake infinite lifetime callback and will instead rely upon fileserver allocated callbacks.
"fs makemount" is supposed to create a mountpoint even if the target volume does not exist or is inaccessible. v2021.05 introduced support for creating mountpoints which kafs which broke the ability to create mountpoints to non-existent volumes.

v2021.05-60 (6 March 2025)

The cache manager retries fileserver RPCs up to 100 times when the fileserver replies with VBUSY, VOFFLINE or VSALVAGE. The cache manager fails the application request after the limit is reached. Prior to this release, the failure would result in the cache manager returning an ENODEV error to the application. But ENODEV indicates that the required volume does not exist and the fileserver has not indicated that the volume does not exist. The cache manager will now return EWOULDBLOCK to the application because the volume might be temporarily inaccessible.
Many OpenAFS and AuriStorFS fileservers can return VNOVOL as a side effect of a race between a volume going offline so that control can be given to the Volume Management Service (volserver) and an incoming RPC requesting that the volume be attached. A VNOVOL can also be returned because the volume was temporarily deleted from the vice partition prior to being replaced by a new clone, or prior to the fileserver marking the volume group as having been moved to another fileserver.
The fileserver replying to an RPC with VNOVOL cannot be trusted to mean that the volume has been permanently removed unless it is matched by the Location Service reporting the Volume Location (VL) entry has been deleted. From now on when a VNOVOL response is received and the VL entry exists, the cache manager will return EWOULDBLOCK instead of ENODEV to the application.
Prior to this release, any VL error code returned from the Location Service would be interpreted as meaning that the requested VL entry does not exist. Starting with this release only the VL_NOENT error will be interpreted as the VL entry does not exist.
Restrict the internal use of the ENOENT error code so the only use of ENOENT is to indicate that a directory entry does not exist. Previously, ENOENT could be returned if a disk cache entry could not be allocated or found. A disk cache related ENOENT returned when processing a directory might have been confused for the requested directory entry not existing.
Restrict the internal use of the ENODEV error code so that the only meaning of ENODEV is that the required volume does not exist. Previously, ENODEV could be returned due to transient errors when failing to contact the Location Service.

v2021.05-59 (13 February 2025)

The fix for garbage collection of idle unixuser objects introduced in v2021.05-56 was incomplete. As a result, Rx client connections might not be garbage collected.

v2021.05-57 (9 February 2025)

No new changes for macOS.

v2021.05-56 (7 February 2025)

Minor bug fixes to the Rx RPC implementation
The fix for garbage collection of idle unixuser objects introduced in v2021.05-55 was incomplete. As a result, Rx client connections might not be garbage collected.

v2021.05-55 (23 January 2025)

Minor bug fixes to the Rx RPC implementation
During garbage collection of idle unixuser objects, any Rx client connections associated with the unixuser (uid or pag) that were in use by another thread would not have their cache manager reference dropped. As a result the Rx client connections could never be garbage collected which is problematic on systems which churn through many unixusers. The behavior has been changed so that unixusers objects with in use Rx connections will not be freed. This permits future attempts to garbage collect unixusers to do so only after the Rx connections can also be freed.
Although this misbehavior has been present for decades, it is only since v2021.05-52 that the problem has been observed. It is possible that the changes to reduce the holding of the GLOCK has created a window of opportunity for connections to be simultaneously in use during unixuser garbage collection.

v2021.05-54 (19 January 2025)

The UNIX cache manager changes continue AuriStor's efforts to address a report that was received at the end of September. The end user site uses an OpenAFS cell to distribute software for execution by IBM Spectrum LSF (load sharing facility) managed compute nodes. Each of the approximately one dozen compute nodes is configured with approximately 100 processor cores and 1TB of ram. Each job scheduled by LSF runs in a newly allocated Process Authentication Group (PAG). Some jobs run for hours or days; others complete in a few seconds. LSF has been observed scheduling more than 3000 jobs in under a minute with peak counts of cache manager user objects at 1300. Everything ran well until it didn't. After some unexplained trigger event, all of the remaining free memory on the system would be consumed in about an hour causing the system to panic.

Over the last five months AuriStor has diligently worked to develop theories, optimize the AuriStorFS UNIX cache manager to address each theory; then move on to subsequent ideas once prior theories were disproved. Over this time we have replaced the memory management lifecycle for access rights entries and rx events. We have made it possible for Linux permission checks to be processed without acquiring the GLOCK in the majority of access patterns thereby increasing parallelism. The lifecycle management of PAG allocation and user entity tracking has been rewritten to avoid the GLOCK. The Linux inode lifecycle management has been rewritten which some sites attribute to elimination of swapping on their systems. All of these changes increased throughput and parallelism within and without the cache manager. However, none of them prevented the memory consumption once the unknown trigger event occurred.

It is with great pleasure that we can report that the trigger event and the source of the memory consumption have been identified. The trigger is a damaged Rx packet header which causes the AuriStor Rx stack to place an Rx connection into an RX_PROTOCOL_ERROR error state when there is an in-flight call to an OpenAFS fileserver. The memory consumption is due to the infinite loop that occurs when an Rx connection is in an error state and the error code is not one of the error codes which forces replacement of the Rx connection. Within each pass of the loop a structure is allocated to track the history of per-server errors so they can be logged when hard mount mode is enabled for the cell. AuriStor Rx permits up to 12 calls on each Rx connection. As a result there can be up to a dozen processes each spinning and allocating memory when the Rx connection is placed into the error state.

Days before Christmas 2023 AuriStor identified a race condition which can result in damaged Rx DATA packets being written to the network. Initial transmission of Rx DATA packets can be sent either from an application thread or an Rx listener. Packet retransmission occurs either in an Rx listener thread or in the Rx event thread. The race permits a second thread to modify the Rx packet headers while the sendmsg() syscall from the original transmission is executing. This race is possible because the original thread might exhaust its timeslice during the sendmsg() call and it might be many milliseconds before it is rescheduled on a busy system. In the meantime, packet retransmissions might be initiated either by an Rx listener thread processing an ACK packet or the Rx event thread processing a timeout event. If the initial packet transmission completes while the Rx headers are being rewritten the resulting packet sent on the wire might be garbage. AuriStor observed that if the modified packets were jumbograms that silent data corruption could occur so we disabled all use of jumbograms in v2021.05-35 We also notified both IBM and OpenAFS of the problem.

v2021.05-53 modified the RPC result analyze logic to force the replacement of Rx connections in an error state except when the error signals a token expiration. When a token is expired there is no benefit to replacing the connection until new tokens have been acquired. This release further ensures that infinite memory allocation cannot occur by tracking only the most recent error per fileserver; and only does so if hard mount mode is enabled for the cell.

Processes that issued syscalls which became stuck in a RPC retry loop could not be interrupted. In addition to the unintentional retry loops there are intentional retry loops such as those that occur because a fileserver repeated fails the call with a “VBUSY” error. “VBUSY” originally meant that the required volume was temporarily in use by the volume management service (“volserver”) and would be back in service soon so please wait. Beginning with IBM AFS 3.4 the error code was used to force a retry for reasons having nothing to do with the requested volume’s availability. IBM AFS 3.4 introduced the concept of multi-homed clients which meant that the fileserver could no longer track the identity of a cache manager by its IPv4 address. Instead, each cache manager randomly generates a UUID which the fileserver must query each time an incoming RPC is received on a previously unrecognized Rx connection. An Rx connection might be unrecognized because it’s new, or because the client’s IP address changed, or because the fileserver garbage collected its state information. Regardless of the reason, if a call is received and there is no Rx connection binding to a cache manager UUID, then the fileserver application worker thread to which the call was assigned must issue an outgoing RPC to the client’s RXAFSCB service. Each client has at most one worker thread. Prior to this release, every incoming RXAFSCB call required acquisition of the GLOCK before it could execute. If the GLOCK is unavailable, the RXAFSCB thread blocks and no other pending RXAFSCB calls can be processed. If a RXAFSCB call blocks for too long, the fileserver timeouts trigger and the fileserver replies to the incoming RXAFS call with “VBUSY”. “VBUSY” can also be sent if the fileserver’s quota on the number of calls from the same endpoint is exceeded.

The v2021.05-54 release removes the GLOCK requirement for many RXAFSCB RPCs including all of the UUID query RPCs and all whole volume callback RPCs. Callback RPCs for individual vnodes must still acquire the GLOCK. However, it is hoped that these changes will significantly reduce the time required to process the queries generated due to unrecognized Rx connections. The AuriStor Rx stack permits a dozen calls to be issued on a single logical connection which the fileserver sees as three unique Rx connections from the same endpoint.

Each Unix UID or PAG ID must use an independent Rx connection to each fileserver. On a system which processes /afs syscalls for 1200 UID or PAG IDs in a minute, there is the potential for 3600 Rx connections per fileserver. When the fileserver is OpenAFS, this quickly triggers per endpoint throttling and generates VBUSY storms which can result in unlucky client processes experiencing starvation. With this release, starving processes can be interrupted and RXAFSCB UUID queries can be completed without blocking.

AuriStor believes this release will provide significant benefits for large scale compute nodes, web servers and time sharing systems.

v2021.05-53 (4 January 2025)

v2021.05-52 switched the memory allocation strategy for rx_event objects from a global free list to reliance upon the operating system's memory allocator to allocate individual objects which are freed upon rx_event completion or cancelation. Reliance upon the system allocator introduces the risk of allocation failures due to out of memory conditions. If allocation of an rx_event object fails, a reference count leak on the associated rx_connection or rx_call object can occur preventing garbage collection of the object.
This release handles rx_event allocation failures:
- delayed ACK or delayed ABORT events are replaced by immediate transmission of the ACK or ABORT
- other cases fail the call or connection with an RX_OUT_OF_MEMORY error which is translated on the wire to RX_WIRE_CALL_DEAD (-1).
After executing a call to a fileserver, if the Rx connection is in an error state for any reason other than expired tokens, then schedule the Rx connection to be replaced the next time a call is attempted. All calls issued on an Rx connection that is already in an error state fail immediately preventing further communication with the associated fileserver from the UID/PAG.
Rx connections in an expired token error state are not replaced until new tokens are set.
Rx connections can be placed into an error state either due to local or remote failures. Such failures are often generated by the in-use security class (RXKAD/RXGK) but might be due to an out-of-memory condition, system clock advancement or reversal, or even application errors.
This release includes internal code changes to the management of servers, connections and access rights which will be used by a future release to separate PAG ID allocation from the system's UID namespace.
If 'afsd' is configured with a keytab, receipt of a PoweredOn event will force an attempt to acquire system tokens.
On supported macOS versions, enforce the use of code signing identities to authenticate mach messages involving the AFSBackgrounder and the AFSPreference panel.

v2021.05-52 (5 December 2024)

The v2021.05-50 replacement of the linked list implementation of the vcache access cache with a btree and last lookup cache introduced a memory corruption bug which is fixed in this release.
Eliminate the GLOCK requirement for the collection of function call and RPC statistics collection.
Replace small space allocation with a GLOCK free allocator.
Reimplement the cache manager tracing logging routines so they no longer require the GLOCK.
The Rx networking stack schedules many different types of time sensitive events associated with individual packet retransmission, delayed ack and abort processing, path mtu detection, keep alives, and other categories. Prior to this release, all memory allocated to Rx events objects would not be freed to the operating system until the userspace process terminated or the kernel module was unloaded.
Beginning with this release, instead of having a single, global, free list for rxevents, the system allocator is trusted to be able to allocate memory quickly. This simplifies the Rx implementation, and should be more efficient, as most system allocators will have per thread or core slabs, removing the contention of the global lock protecting the rxevent free list. In addition, the change protects against a network event allocating memory that cannot be readily freed.
On non-Total Store Ordering (TSO)architectures, relaxed atomics used as reference counters do not ensure completion of all prior stores or recency of later loads. This release replaces all read-modify-write atomics to be acquire-release, which, among other things, ensures that only the intended ordering is possible.

v2021.05-51 (26 November 2024)

Fix a vcache (aka inode) reference leak that can occur during an access rights query for a non-directory object if the neither the calling process nor the cache manager possesses tokens for the cell containing the object. The leak was introduced in v2021.05-45.
When scanning the token list for each user searching for expired tokens the user spinlock must be held to prevent races. The updated user management implementation pulled into v2021.05-45 failed to hold the spinlock.
During cache manager module shutdown reset global structures to the module initialization state. Doing so improves the likelihood that the module can be reinitialized.

v2021.05-50 (23 November 2024)

The linked list implementation of the vcache access rights cache was replaced with a btree and a last lookup cache. The btree design is more efficient for lookups and removals, and scales better as the number of uids and/or process authentication groups (pags) accessing /afs increases.
Prune access rights for expired users. Doing so reduces the cost of subsequent lookups.
Cache the access rights obtained for a uid|pag via the "anyAccess" field. Previously, a uid|pag that obtained its rights from the "anyAccess" field had to perform and fail a rights lookup on each and every vcache access.
When setting or clearing tokens for a single cell, no longer invalidate the access rights cached for the uid|pag for all cells. Instead, only invalidate the access rights cached for the cell whose tokens were set or cleared.

v2021.05-49 (16 November 2024)

The output of "tokens" command failed to report yfs-rxgk tokens was broken starting in v2021.05-46

v2021.05-48 (12 November 2024)

Preallocated buffer overflows in XDR responses (CVE-2024-10397)
The AuriStorFS and AFS3 RPC suites rely upon Sun RPC XDR to marshal binary data structures for network transfer. The AuriStor XDR implementation is derived from Sun Microsystems' Sun RPC code base. The Sun RPC XDR API permits memory for output parameters to (optionally) be preallocated which can result in various classes of memory corruption and/or memory leaks in RPC initiator processes.
The AuriStorFS v2021.05-48 release introduces additional data length validation checks within the AuriStor XDR implementation and prohibits the use of preallocated memory for string output parameters or fields. All cache managers, servers and command line tools are modified by these changes.

v2021.05-46 (28 October 2024)

Cache Manager:
- Prevent a kernel memory leak when server preferences are set via the yfs-client.conf [afsd] configuration or via "fs setserverprefs".
- Directory enumeration of a truncated directory now returns an error instead of assuming the end of the directory has been reached.
- Since AFS 3.0, the Unix cache manager has used the root identity credentials to create anonymous outgoing connections to the location service and each fileserver. However, if uid 0 is assigned a token, then those Rx connections will no longer be anonymous. Beginning with this release anonymous outgoing connections are always created with the NOPAG identity (uid 0xffffffff) instead of the root identity.
- When establishing an outgoing rxgk connection, do not fallback to the systemuser's credentials if the user's credentials resulted in a fatal error. Falling back to the systemuser's credentials can result in inappropriate use of an anonymous connection.
- Improved access rights cache correctness for YFS servers
  In prior releases, the access check logic used the file rights for any files fetched from an AuriStorFS fileserver. For files fetched from an AFS-3 fileserver (and, historically, for all files), it used the directory rights, with the (a)dmin right from the file mixed in. The (a)dmin right on a non-directory indicates that the object is owned by the authenticated user.
  This approach has some issues when combined with the access rights cache, and current fileserver callback behavior. On an AuriStorFS file server, the rights on a non-directory may be determined by the rights granted on its parent directory or, with per-file ACLs, those granted on the object itself. The fileserver will only break a non-directory's callback when a per-file ACL is changed - changing a directory ACL will not break callbacks on files within that directory. This means that changing a directory ACL will not invalidate access rights cache entries on files in tha directory, even if the effective ACL on this files has changed, and the cached rights are no longer correct.
  This release works around this by adding a new function which returns the access rights for a file hosted on an AuriStor fileserver. It uses the parent vnode information to locate the parent directory. If the parent directory isn't in the cache, or it doesn't have a valid callback, or if it has been changed since the file's access rights were cached, it clears the current access rights. Files without a parent directory must have per-file ACLs, and so their cached rights can be safely used.
  Note that files with parent vnodes may still have per-file ACLs, and that the breadcrumbing performed by the client may add parent vnode fields to vnodes which don't have them provided by the fileserver. Such vnodes may have their cached access rights cleared more frequently than necessary.
- Add a new mechanism for caching access rights within the vcache structure. This cache is protected via a vcache specific spinlock, and can be accessed without holding the GLOCK.
  This new cache mechanism returns the memory associated with cached rights back to the kernel's slab free memory pool instead of adding the unused rights structures to a cache manager managed free list. The previous cache implementation never returned allocated memory to the kernel. Instead, invalidated access rights were appended to a free access rights queue for later reuse.
- When a volume is accessed via multiple mountpoints a choice must be made regarding which mountpoint is considered to be the active (or parent) mountpoint. This release alters the behavior such that the active mountpoint is set every time a mountpoint is traversed.
  This behavior is easier to understand and is more likely to provide the expected result for a single process that repeatedly accesses volumes from multiple mountpoints. However, it can result in unexpected results when multiple processes are traversing multiple mountpoints in parallel without any synchronization.

v2021.05-45 (Not released)

v2021.05-44a (18 September 2024)

Authentication:
- AuriStorFS v2021.05-44 included an updated version of the Heimdal Kerberos framework used by AuriStorFS when acquiring yfs-rxgk and rxkad authentication tokens. The updated Heimdal included a bug which disabled the use of DNS SRV records for KDC discovery and DNS TXT records for realm discovery. As a side effect token acquisition might fail with an unable to reach any KDC in realm error. This is fixed in v2021.05-44a.

v2021.05-44 (17 August 2024)

Cache Manager:
- Since v0.192 the cache manager has failed acquire the global lock when upgrading a shared-lock to a write-lock during the execution of a background cache chunk file truncation.
Authentication:
- Neither MIT nor Heimdal gssapi nor their gss mechanisms consistently initialize the output 'minorStatus' parameter. Various functions can return either success or failure majorStatus values with minorStatus unassigned. As a result, stack garbage will be used when generating error messages. From now on libyfs_acquire will always initialize the minorStatus output variable to zero before calling into the gssapi library.
Command Parser:
- No longer accept the token "-" as a switch which eventually fails with a CMD_UNKNOWNSWITCH error. Instead, process the token as a data value.
- Optimize the processing of the loop which processes "source" command input.
- If the source command input file is "-", read from stdin.

v2021.05-41 (26 June 2024)

Rx Networking (libyfs_rx):
- A race during event creation can lead to the freeing of the event while its still in use.
- RFC1122 says that Net and Host unreachable ICMP errors might be transient and should therefore not be treated as fatal. There is no such language for the equivalent ICMPV6 errors. However, in practice ICMP6_DST_UNREACH_NOROUTE, ICMP6_DST_UNREACH_BEYONDSCOPE, and ICMP6_DST_UNREACH_ADDR can be transient.
  Linux has considered these ICMPV6 destination unreachable errors as non-fatal going back at least as far as the initial git repository commit.
  AuriStor Rx has always treated these as fatal errors which results in immediate termination of in-flight calls when received. Even if the network route corrects itself before the call timeout period expires. This release mirrors the Linux behavior and makes these errors non-fatal.
Cache Manager:
- For the first time the cache manager can detect the deletion of a volume and handle the creation of a new volume with the same name but a different volume id.
- If the location service reports the deletion of a volume, invalidate all mount points to that volume.
- RXAFS_GetCapabilities RPC failures should not be treated as a fatal error preventing failover to another replica site.
Authentication ("libyfs_acquire") used by aklog, vos, pts, bos, afsio:
- rxkad_k5 token acquisition krb5 ccache management. This release altered the krb5 credential cache management strategy once again to work around different bugs in MIT krb5 and Heimdal.
- New ACQUIRE_ERR_CRED_EXPIRED error code introduced to represent the case when a request for a service credential returns one that is already expired.
Command parser (libyfs_cmd):
- When parsing configuration files there is a depth limit of ten active inclusions. This limit was improperly enforced as a limit of ten included files instead of a depth of ten included files. As of this release it is now possible to populate an includedir directory with any number of .conf files.

v2021.05-40

Not released.

v2021.05-39 (20 May 2024)

Parallel Random Number Generation:
- AuriStorFS processes rely upon the krb5_generate_random() and RAND_bytes() functions to obtain random bytes for cryptographic operations and random counters. krb5_generate_random() internally acquires a mutex to protect internal state information. This mutex has become a significant barrier to the encryption and checksumming of Rx packets with both yfs-rxgk and rxkad.
  
  This release replaces general use of krb5_generate_random() and RAND_bytes() with a per-thread ChaCha20 CS-PRNG. This avoids the acquisition of a global mutex and permits increased parallelism on multi-core systems.
Rx Networking (libyfs_rx):
- The Rx network stack schedules a garbage collection operation to execute once per minute. This operation enforces call timeouts, destroys idle connections and destroys idle peers. The operation has historically been performed by the Rx event thread which is already responsible for performing actions in response to call RTOs, sending NAT Ping and keep-alive packets, and retrying connection challenge and reachability checks.
  
  The time complexity of the garbage collection operation is determined by the number of calls, connections, and peers. The busier the Rx endpoint the more work must be performed during each garbage collection run and the longer it takes to complete. While garbage collection is active other events cannot be processed which can interfere with the proper flow control of active calls.
  
  As with all Rx events, the garbage collection event is scheduled to execute at an absolute clock time. If the system clock drifts (or is administratively set) backwards garbage collection will not be performed until the clock catches up with the scheduled time.
  
  Another responsibility of the garbage collection procedure is to terminate calls if the system clock drifted backwards by five minutes or longer. However, when the clocked drifts backwards garbage collection is not performed until the clock has advanced beyond the point where calls require termination. As a result, calls are not terminated due to backwards clock drift and they can stall.
  
  This release re-implements the garbage collection procedure using a dedicated thread and relative waits. This change ensures that the garbage collection procedure will not prevent the execution of call related events and permits calls to be terminated when large backward clock drifts are detected.
Disk Cache Management:
- Since IBM AFS 3.5, the cache has been considered "too full" even if there exist cache files have been discarded but not yet truncated. When the cache is "too full" most operations that write to the cache will block until truncation of discarded cache files has been performed which results in unnecessary delays. This release fixes the cache such that that discarded but not yet truncated cache files do not block write operations.
- This release permits the cache truncation daemon thread to exit sooner if the cache manager is shutting down.
Improved failover when the RXGK service (co-located with each vlserver) fails to issue tokens. The failures might be the result of misconfiguration, an inability to read keys or loss of Ubik quorum.

v2021.05-38 (29 February 2024)

As with other AuriStorFS releases since the beginning of 2024, this release includes additional improvements to the Rx RPC implementation which are related to the possibility of silent data corruption when Rx jumbograms are in use. Prior releases disabled the negotiation of Rx jumbograms such that the v2021.05-37 Rx peer will refuse to send Rx jumbograms and will request that the remote peer does not send them. However, a bad actor could choose to send Rx jumbograms even though they were asked not to. v2021.05-38 introduces additional protections to ensure that a corrupt Rx jumbogram is dropped instead of being accepted.

The v2021.05-38 Rx RPC implementation also includes two optimizations. First, when Rx initiators complete a call they will no longer send an extra ACK packet to the Rx acceptor of the completed call. The sending of this unnecessary ACK creates additional work for the server which can result in increased latency for other calls being processed by the server.

Second, all AuriStor Rx services require a reach check for incoming calls from Rx peers to help protect against Distributed Reflection Denial of Service (DRDoS) attacks and execution of RPCs when the response cannot be delivered to the caller. A new reach check is required for each new call that arrives more than 60 seconds after the prior reach check completed. v2021.05-38 Rx considers the successful acknowledgment of a response DATA packet as a reach check validation. With this change reach checks will not be periodically required for a peer that completes at least one call per 60 seconds. A 1 RTT delay is therefore avoided each time a reach check can be avoided. In addition, reach checks require the service to process an additional ACK packet. Eliminating a large number of reach checks can improve overall service performance.

The final Rx RPC change in this release is specific to kernel implementations. Prior releases restricted the frequency of executing time scheduled Rx events to a granularity no smaller than 500ms. As a result an RTO timer event for a lost packet could not be shorter than 500ms even if the measured RTT for the connection is significantly smaller. The minimum RTO for a connection in AuriStor Rx is 200ms. The inability to schedule shorter timeouts impacts recovery from packet loss.

v2021.05-37 (5 February 2024)

Rx improvements:
- The v2021.05-36 release permanently disabled all use of Rx jumbograms due to a risk of silent data corruption. However, when advertising the number of acceptable datagrams in the ACK trailer a missing htonl() set the value to 16777216 instead of 1 on little-endian systems.
- When sending a PING ACK as a reachability test, ensure that the previousPacket field is properly assigned to the largest accepted DATA packet sequence number instead of zero.
- Replace the initialization state flag with two flags. One that indicates that Rx initialization began and the other that it succeeded. The first prevents multiple attempts at initialization after failure. The second prevents shutdown from accessing uninitialized structures if initialization failed.
Cache Manager Improvements:
- No longer refuse to start if both the 'cachedir' and 'memcache' options are present in the configuration file.
- New variable to store the maximum number of cache blocks used. which is accessible via /proc/fs/auristorfs/cache/blocks_used_max.

v2021.05-36 (10 January 2024)

Rx improvements:
- Permanently disable all use of Rx jumbograms due to a risk of silent data corruption.
  
  Ever since OpenAFS 1.0, and possibly before, a race condition has existed when Rx transmits packets. As the rx_call.lock is dropped when starting packet transmission, there is no protection for data that is being copied into the kernel by sendmsg(). It is critical that this packet data is not modified by another thread. However, races exist between the application, listener, and event threads that can lead to retransmissions starting whilst an original transmission is still in progress. This can lead to the packet headers being overwritten, and either the original transmission, the retransmission or both sending corrupt data to the peer.
  
  This corruption can affect the packet serial number or packet flags. It is particularly harmful when the packet flags are corrupted, as this can lead to multiple Rx packets which were intended to be sent as Rx jumbograms being delivered and misinterpreted as a single large packet. The eventual result of this depends on the Rx security class in play, but it can cause decrypt integrity errors (rxgk:crypt and rxgk:auth) or corruption of the data stream (rxnull, rxgk:clear or rxkad:auth).
  
  All AuriStorFS servers, OpenAFS 1.6 or later servers, and the Windows cache manager have been shipped with Rx jumbograms disabled by default. The UNIX cache managers however are shipped with jumbograms enabled. There are many AFS cells around the world that continue to deploy OpenAFS 1.4 or earlier fileservers which continue to negotiate the use of Rx jumbograms.
  
  It is worth noting that all AuriStorFS v0.198 and later fileservers and cache managers implement explicit checks that will recognize the corrupted application data stream and prevent corrupted file content from being stored either into an AFS vnode's backing store in the volume's object store or the cache manager's AFS cache. OpenAFS cache managers and fileservers do not have these protections.
  
  With Rx jumbograms disabled the maximum number of Rx packets in a datagram is reduced from 6 to 1; the maximum number of send and receive datagram fragments is reduced from 4 to 1; and the maximum advertised MTU is restricted to 1444 - the maximum rx packet size prior to the introduction of jumbograms in IBM AFS 3.5.
- If the rx call flow state transitions from either the RECOVERY or RESCUE states to the LOSS state as a result of an RTO resend event while writing packets to the network, cease transmission of any new DATA packets if there are packets in the resend queue.
- When the call flow state is LOSS and all packets in the resend queue have been retransmitted and yet the recovery point has not been reached, then permit new DATA packets to be sent in order to maintain a full congestion window.
- Add a safety check to prevent the estimated RTT from underflowing when the actual roundtrip time is smaller than 125us.
- Fix the computation of the padding required for rxgk encrypted packets. This bug results in packets sending 8 bytes fewer per packets than the network permits. This bug accidentally prevented the construction of Rx jumbograms when a call is protected by rxgk:crypt.
- Replace the random number generator with a more security source of random bytes.

v2021.05-33 (27 November 2023)

Rx improvements:
- Not all calls transfer enough data to be able to measure a smoothed round-trip time (SRTT). Calls which are unable to compute a SRTT should not be used to update the peer host RTO value which is used to initialize the RTO for subsequent calls.
  
  Without this change, a single DATA packet call will cause the peer host RTO to be reduced to 0ms. Subsequent calls will start with a RTO value of MAX(0, rxi_minPeerTimeout) where rxi_minPeerTimeout defaults to 200ms. If the actual measured RTO is greater than 200ms, then initial RTO will be too small resulting in premature triggering of the RTO timer and the call flow state entering the loss phase which can significantly hurt performance.
- Initialize the peer host RTO to rxi_minPeerTimeout (which defaults to 200ms) instead of one second. Although RFC6298 recommends the use of one second when no SRTT is available, Rx has long used the rxi_minPeerTimeout value for other purposes which are supposed to be consistent with initial RTO value. It should be noted that Linux TCP uses 200ms instead of one second for this purpose.
- If associating a security class with an Rx connection fails immediately place the Rx connection into an error state. A failure might occur if the security class is unable to access valid key material.
- If an incoming Rx call requires authentication and the security class is unable to successfully generate a challenge, put the incoming Rx connection into an error state and issue an abort to the caller.
- If an incoming Rx call requires authentication and the security class is able to generate a challenge but the challenge cannot be returned to Rx, then treat this as a transient error. Do not acknowledge the incoming DATA packet and do not place the Rx connection into an error state. An attempt to re-issue the challenge will be performed when the DATA packet is retransmitted.
- If an Rx call is terminated due to the expiration of the configured connection dead time, idle dead time, hard dead time, or as a result of clock drift, then send an ABORT to the peer notifying them that the call has been terminated. This is particularly important for terminated outgoing calls. If the peer does not know to terminate the call, then the call channel might be in use when the next outgoing call is issued using the same call channel. If the next incoming call is received by an in-use call channel, the receiver must drop the received DATA packet and return a BUSY packet. The call initiator will need to wait for a retransmission timeout to pass before retransmitting the DATA packet. Receipt of BUSY packets cannot be used to keep a call alive and therefore the requested call is at greater risk of timing out if the network path is congested.
aklog and krb5.log (via libyfs_acquire):
- If the linked Kerberos library implements krb5_cc_cache_match() and libacquire has been told to use an explicit principal name and credential cache, the Kerberos library might return KRB5_CC_NOTFOUND even though the requested credential cache is the correct one to use. This release will not call krb5_cc_cache_match() if the requested credential cache contains the requested principal.
Cell Service Database (cellservdb.conf):
- cellservdb.conf has been synchronized with the 31 Oct 2023 update to the grand.central.org CellServDB file.

v2021.05-32 (9 October 2023)

No significant changes for macOS compared to v2021.05-31

v2021.05-31 (25 September 2023)

New platform:
- macOS 14 Sonoma
macOS 14 Sonoma:

AuriStorFS v2021.05-29 and later installers for macOS 13 Ventura are compatible with macOS 14 Sonoma and do not need to be removed before upgrading to macOS 14 Sonoma. Installation of the macOS 14 Sonoma version of AuriStorFS is recommended.

Cache Manager:
- If an AuriStorFS cache manager is unable to use the yfs-rxgk security class when communicating with an AuriStorFS fileserver, it must assume it is IBM AFS 3.6 or OpenAFS and upgrade it to AuriStorFS if an upgrade probe returns a positive result. Once a fileserver's type is identified as AuriStorFS the type should never be reset; even if communication with the fileserver is lost or the fileserver restarts.
  
  If an AuriStorFS fileserver is replaced by an OpenAFS fileserver on the same endpoint, then the UUID of the OpenAFS must be different. As a result, the OpenAFS fileserver will be observed as distinct from the AuriStorFS fileserver that previously shared the endpoint.
  
  Prior to this release there were circumstances in which the cache manager discarded the fileserver type information and would fail to recognize the fileserver as an AuriStorFS fileserver when yfs-rxgk could not be used. This release prevents the cache manager from resetting the type information if the fileserver is marked down.
- If a fileserver's location service entry is updated with a new uniquifier value (aka version number), this indicates that one of the following might have changed:
  1. the fileserver's capabilities
  2. the fileserver's security policy
  3. the fileserver's knowledge of the cell-wide yfs-rxgk key
  4. the fileserver's endpoints
  Beginning with this release the cache manager will force the establishment of new Rx connections to the fileserver when the uniquifier changes. This ensures that the cache manager will attempt to fetch new per-fileserver yfs-rxgk tokens from the cell's RXGK service, enforce the latest security policy, and not end up in a situation where its existing tokens cannot be used to communicate with the fileserver.
aklog:
- Fix incorrect output when populating the server list for a service fails. The stashed extended error explaining the cause of the failure was not displayed.
- If a cell has neither _afs3-prserver._udp. DNS SRV records nor AFSDB records, the lookup of the cell's protection servers would fail if there is no local cell configuration details. The fallback to use _afs3-vlserver._udp. DNS SRV records did not work. This is corrected in this release.

v2021.05-30 (6 September 2023)

Do not mark a fileserver down in response to a KRB5 error code.
fs cleanacl must not store back to the file server a cleaned acl if it was inherited from a directory. Doing so will create a file acl.
Correct the generation of never expire rxkad_krb5 tokens from Kerberos v5 tickets which must have a start time of Unix epoch and an end time of 0xFFFFFFFF seconds. The incorrectly generated tokens were subject to the maximum lifetime of 30 days.
Correct the generation of the yfs-rxgk RESPONSE packet header which failed to specify the key version generation number used to encrypt the authenticator. If the actual key version is greater than zero, then the authenticator would fail to verify.
Enforce a maximum NAT ping period of 20s to ensure that NAT/PAT/firewall rules due not expire while Rx RPCs are in-flight.

v2021.05-29 (26 June 2023)

Execution of fs commands such as examine, whereis, listquota, fetchacl, cleanacl, storeacl, whoami, lsmount, bypassthreshold and getserverprefs could result in memory leaks by the AuriStorFS kernel extension.

v2021.05-27 (1 May 2023)

Fixes for bugs in vos introduced in v2021.05-26.

v2021.05-26 (17 April 2023)

Fixed a potential kernel memory leak when triggered by fs examine, fs listquota, or fs quota.
Increased logging of VBUSY, VOFFLINE, VSALVAGE, and RX_RESTARTING error responses. A log message is now generated whenever a task begins to wait as a result of one of these error responses from a fileserver. Previously, a message was only logged if the volume location information was expired or discarded.
Several changes to optimize internal volume lookups.
Faster failover to replica sites when a fileserver returns RX_RESTARTING, VNOVOL or VMOVED.
rxdebug regains the ability to report rx call flags and rx_connection flags.
The RXRPC library now terminates calls in the QUEUED state when an ABORT packet is received. This clears the call channel making it available to accept another call and reduces the work load on the worker thread pool.
Fileserver endpoint registration changes no longer result in local invalidation of callbacks from that server.
Receipt of an RXAFSCB_InitCallBackState3 RPC from a fileserver no longer resets the volume site status information for all volumes on all servers.

v2021.05-25 (28 December 2022)

The v2021.05-25 release includes further changes to RXRPC to improve reliability. The changes in this release prevent improper packet size growth. Packet size growth should never occur when a call is attempting to recover from packet loss; and is unsafe when the network path's maximum transmission unit is unknown. Packet size growth with be re-enabled in a future AuriStorFS release that includes Path MTU detection and the Extended SACK functionality.
Improved error text describing the source of invalid values in /etc/yfs/yfs-client.conf or included files and directories.

v2021.05-24 (25 October 2022)

New Platform: macOS 13 (Ventura)

v2021.05-22 (12 September 2022) and v2021.05-21 (6 September 2022)

RX RPC
- If receipt of a DATA packet causes an RX call to enter an error state, do not send the ACK of the DATA packet following the ABORT packet. Only send the ABORT packet.
- AuriStor RX has failed to count and report the number of RX BUSY packets that have been sent. Beginning with this change the sent RX BUSY packet count is once again included in the statistics retrieved via rxdebug server port -rxstats.
- Introduce minimum and maximum bounds checks on the ACK packet trailer fields. If the advertised values are out of bounds for the receiving RX stack, do not abort the call but adjust the values to be consistent with the local RX RPC implementation limits. These changes are necessary to handle broken RX RPC implementations or prevent manipulation by attackers.

New to v2021.05-20 (15 August 2022) and v2021.05-19 (13 August 2022)

RX RPC
- Include the DATA packet serial number in the transmitted reachability check PING ACK. This permits the reachability test ACK to be used for RTT measurement.
- Do not terminate a call due to an idle dead timeout if there is data pending in the receive queue when the timeout period expires. Instead deliver the received data to the application. This change prevents idle dead timeouts on slow lossy network paths.
- Fix assignment of RX DATA, CHALLENGE, and RESPONSE packet serial numbers in macOS (KERNEL). Due to a mistake in the implementation of atomic_add_and_read the wrong serial numbers were assigned to outgoing packets.

New to v2021.05-18 (12 July 2022)

Cache Manager
- Prevent a kernel memory leak of less than 64 bytes for each bulkstat RPC issued to a fileserver. Bulkstat RPCs can be frequently issued and over time this small leak can consume a large amount of kernel memory. Leak introduced in AuriStorFS v0.196.
- The Perl::AFS module directly executes pioctls via the OpenAFS compatibility pioctl interface instead of the AuriStorFS pioctl interface. When Perl::AFS is used to store an access control list (ACL), the deprecated RXAFS_StoreACL RPC would be used in place of the newer RXAFS_StoreACL2 or RXYFS_StoreOpaqueACL2 RPCs. This release alters the behavior of the cache manager to use the newer RPCs if available on the fileserver and fallback to the deprecated RPC. The use of the deprecated RPC was restricted to use of the OpenAFS pioctl interface.
RX RPC
- Handle a race during RX connection pool probes that could have resulted in the wrong RX Service ID being returned for a contacted service. Failure to identify that correct service id can result in a degradation of service.
- The Path MTU detection logic sends padded PING ACK packets and requests a PING_RESPONSE ACK be sent if received. This permits the sender of the PING to probe the maximum transmission unit of the path. Under some circumstances attempts were made to send negative padding which resulted in a failure when sending the PING ACK. As a result, the Path MTU could not be measured. This release prevents the use of negative padding.
Preparation for supporting macOS 13 Ventura when it is released in Fall 2022.
Some shells append a slash to an expanded directory name in response to tab completion. These trailing slashes interfered with "fs lsmount", "fs flushmount" and "fs removeacl" processing. This release includes a change to prevent these commands from breaking when presented a trailing slash.

New to v2021.05-17 (16 May 2022)

Cell Service Database Updates
- Update cern.ch, ics.muni.cz, ifh.de, cs.cmu.edu, qatar.cmu.edu, it.kth.se
- Remove uni-hohenheim.de, rz-uni-jena.de, mathematik.uni-stuttgart.de, stud.mathematik.uni-stuttgart.de, wam.umd.edu
- Add ee.cooper.edu
- Restore ams.cern.ch, md.kth.se, italia
Fix parsing of [afsd] rxwindow configuration which can be used to specified a non-default send/receive RX window size. The current default is 128 packets.
RX Updates
- Add nPacketsReflected

Logging improvements

Cache directory validation errors log messages now include the cache directory path.

Log the active configuration path if "debug" logging is enabled.

More details of rxgk token extraction failures.

New to v2021.05-16 (24 March 2022)

RX - Previous releases re-armed the Retransmission Timeout (RTO) each time a new unacknowledged packet is acknowledged instead of when a new leading edge packet is acknowledged. If leading edge data packet and its retransmission are lost, the call can remain in the "recovery" state where it continues to send new data packets until one of the following is true:
. the maximum window size is reached
. the number of lost and resent packets equals 'cwind'
at which point there is nothing left to transmit. The leading edge data packet can only be retransmitted when entering the "loss" state but since the RTO is reset with each acknowledged packet the call stalls for one RTO period after the last transmitted data packet is acknowledged.

This poor behavior is less noticiable with small window sizes and short lived calls. However, as window sizes and round-trip times increase the impact of a twice lost packet becomes significant.

RX - Never set the high-order bit of the Connection Epoch field. RX peers starting with IBM AFS 3.1b through AuriStor RX v0.191 ignore the source endpoint when matching incoming packets to RX connections if the high-order epoch bit is set. Ignoring the source endpoint is problematic because it can result in a call entering a zombie state whereby all PING ACK packets are immediately responded to the source endpoint of the PING ACK but any delayed ACK or DATA packets are sent to the endpoint bound to the RX connection. An RX client that moves from one network to another or which has a NAT|PAT device between it and the service can find themselves stuck.

Starting with AuriStor RX v0.192 the high-order bit is ignored by AuriStor RX peer when receiving packets. This change to always clear the bit prevents IBM AFS and OpenAFS peers from ignoring the source endpoint.

RX - The initial packetSize calculation for a call is altered to require that all constructed packets before the receipt of the first ACK packet are eligible for use in jumbograms if and only if the local RX stack has jumbograms enabled and the maximum MTU is large enough. By default jumbograms are disabled for all AuriStorFS services. This change will have a beneficial impact if jumbograms are enabled via configuration; or when testing RX performance with "rxperf".

New fs whereis -noresolve option displays the fileservers by network endpoint instead of DNS PTR record hostname.

New to v2021.05-15 (24 January 2022)

kernel - fixed YFS_RXGK service rx connection pool leak

New to v2021.05-14 (20 January 2022)

fs mkmount permit mount point target strings longer than 63 characters.
afsd enhance logging of yfs-rxgk token renewal errors.

afsd gains a "principal = configuration option for use with keytab acquisition of yfs-rxgk tokens for the cache manager identity.

kernel - Avoid unnecessary rx connection replacement by racing threads after token replacement or expiration.

kernel - Fix a regression introduced in v2021.05 where an anonymous combined identity yfs-rxgk token would be replaced after three minutes resulting in the connection switching from yfs-rxgk to rxnull.

kernel - Fix a regression introduced in v0.208 which prevented the invalidation of cached access rights in response to a fileserver callback rpc. The cache would be updated after the first FetchStatus rpc after invalidation.

kernel - Reset combined identity yfs-rxgk tokens when the system token is replaced.

kernel - The replacement of rx connection bundles in the cache manager to permit more than four simultaneous rx calls per uid/pag with trunked rx connections introduced the following regressions in v2021.05.

a memory leak of discarded rx connection objects

failure of NAT ping probes after replacement of an connection

inappropriate use of rx connections after a service upgrade failure

All of these regressions are fixed in patch 14.

New to v2021.05-12 (7 October 2021)

fs ignorelist -type afsmountdir in prior releases could prevent access to /afs.

Location server rpc timeout restored to two minutes instead of twenty minutes.

Location server reachability probe timeout restored to six seconds instead of fifty seconds.

Cell location server upcall results are now cached for fifteen seconds.

Multiple kernel threads waiting for updated cell location server reachability probes now share the results of a single probe.

RX RPC implementation lock hierarchy modified to prevent a lock inversion.

RX RPC client connection reference count leak fixed.

RX RPC deadlock during failed connection service upgrade attempt fixed..

New to v2021.05-9 (25 October 2021)

First public release for macOS 12 Monterey build using XCode 13. When upgrading macOS to Monterey from earlier macOS releases, please upgrade AuriStorFS to v2021.05-9 on the starting macOS release, upgrade to Monterey and then install the Monterey specific v2021.05-9 release.

Improved logging of "afsd" shutdown when "debug" mode is enabled.

Minor RX network stack improvements

New to v2021.05-3 (10 June 2021)

Fix for [cells] cellname = {...} without server list.

New to v2021.05 (31 May 2021)

Multi-homed location servers are finally managed as a single server instead of treating each endpoint as a separate server. The new functionality is a part of the wholesale replacement of the former cell management infrastructure. Location server communication is now entirely managed as a cluster of multi-homed servers for each cell. The new infrastructure does not rely upon the global lock for thread safety.

This release introduces a new infrastructure for managing user/pag entities and tracking their per cell tokens and related connection pools.

Expired tokens are no longer immediately deleted so that its possible for them to be listed by "tokens" for up to two hours.

Prevent a lock inversion introduced in v0.208 that can result in a deadlock involving the GLOCK and the rx call.lock. The deadlock can occur if a cell's list of location servers expires and during the rebuild an rx abort is issued.

Add support for rxkad "auth" mode rx connections in addition to "clear" and "crypt". "auth" mode provides integrity protection without privacy.

Add support for yfs-rxgk "clear" and "auth" rx connection modes.

Do not leak a directory buffer page reference when populating a directory page fails.

Re-initialize state when populating a disk cache entry using the fast path fails and a retry is performed using the slow path. If the data version changes between the attempts it is possible for truncated disk cache data to be treated as valid.

Log warnings if a directory lookup operation fails with an EIO error. An EIO error indicates that an invalid directory header, page header, or directory entry was found.

Do not overwrite RX errors with local errors during Direct-I/O and StoreMini operations. Doing so can result in loss of VBUSY, VOFFLINE, UAENOSPC, and similar errors.

Correct a direct i/o code path which could overwrite a fileserver returned error code with a local RXGEN_CC_UNMARSHAL error.

Correct the StoreMini code path which could overwrite a fileserver returned error code with a local RXGEN_CC_UNMARSHAL error.

Ensure the rx call object is not locked when writing to the network socket.

Removed all knowledge of the KERNEL global lock from RX. Acquiring the GLOCK from RX is never safe if any other lock is held. Doing so is a lock order violation that can result in deadlocks.

Fixed a race in the opr_reservation system that could produce a cache entry reference undercount.

If a directory hash chain contains a circular link, a buffer page reference could be leaked for each traversal.

Each AFS3 directory header and page header contains a magic tag value that can be used in a consistency check but was not previously checked before use of each header. If the header memory is zero filled during a lookup, the search would fail producing an ENOENT error. Starting with this release the magic tag values are validated on each use. An EIO error is returned if there is a tag mismatch.

"fs setcrypt -crypt auth" is now a permitted value. The "auth" mode provides integrity protection but no privacy protection.

Add new "aklog -levels " option which permits requesting "clear" and "auth" modes for use with yfs-rxgk.

Update MKShim to Apple OpenSource MITKerberosShim-79.

Report KLL errors via a notification instead of throwing an exception which (if not caught) will result in process termination.

If an exception occurs while executing "unlog" catch it and ignore it. Otherwise, the process will terminate.

New to v2021.04 (22 April 2021)

Primarily bug fixes for issues that have been present for years.

A possibility of an infinite kernel loop if a rare file write / truncate pattern occurs.

A bug in silly rename handling that can prevent cache manager initiated garbage collection of vnodes.

New to v0.209 (13 March 2021)

fs setserverprefs and fs getserverprefs updated to support IPv6 and CIDR specifications.

Improved error handling during fetch data and store data operations.

Prevents a race between two vfs operations on the same directory which can result in caching of out of date directory contents.

Use cached mount point target information instead of evaluating the mount point's target upon each access.

Avoid rare data cache thrashing condition.

Prevent infinite loop if a disk cache error occurs after the first page in a chunk is written.

Network errors are supposed to be returned to userspace as ETIMEDOUT. Previously some were returned as EIO.

When authentication tokens expire, reissue the fileserver request anonymously. If the anonymous user does not have permission either EACCES or EPERM will be returned as the error to userspace. Previously the vfs request would fail with an RXKADEXPIRED or RXGKEXPIRED error.

If growth of an existing connection vector fails, wait on a call slot in a previously created connection instead of failing the vfs request.

Volume and fileserver location query infrastructure has been replaced with a new modern implementation.

Replace the cache manager's token management infrastructure with a new modern implementation.

New to v0.206 (12 January 2021) - Bug fixes

Prevents a possible panic during unmount of /afs.

Improved failover and retry logic for offline volumes.

New to v0.205 (24 December 2020) - Bug fixes

Volume name-to-id cache improvements

Fix expiration of name-to-id cache entries

Control volume name-to-id via sysctl

Query volume name-to-id statistics via sysctl

Improve error handling for offline volumes

Fix installer to prevent unnecessary installation of Rosetta 2 on Apple Silicon

New to v0.204 (25 November 2020) - Bug fix for macOS Big Sur

v0.204 prevents a kernel panic on Big Sur when AuriStorFS is stopped and restarted without an operating system reboot.

introduces a volume name-to-id cache independent of the volume location cache.

New to v0.203 (13 November 2020) - Bug fix for macOS

v0.203 prevents a potential kernel panic due to network error.

New to v0.201 (12 November 2020) - Universal Big Sur (11.0) release for Apple Silicon and Intel

v0.201 introduces a new cache manager architecture on all macOS versions except for High Sierra (10.12). The new architecture includes a redesign of:

kernel extension load

kernel extension unload (not available on Big Sur)

/afs mount

/afs unmount

userspace networking

The conversion to userspace networking will have two user visible impacts for end users:
The Apple Firewall as configured by System Preferences -> Security & Privacy -> Firewall is now enforced. The "Automatically allow downloaded signed software to receive incoming connections" includes AuriStorFS.

Observed network throughput is likely to vary compared to previous releases.

On Catalina the "Legacy Kernel Extension" warnings that were displayed after boot with previous releases of AuriStorFS are no longer presented with v0.201.

AuriStorFS /afs access is expected to continue to function when upgrading from Mojave or Catalina to Big Sur. However, as AuriStorFS is built specifically for each macOS release, it is recommended that end users install a Big Sur specific AuriStorFS package.
AuriStorFS on Apple Silicon supports hardware accelerated aes256-cts-hmac-sha1-96 and aes128-cts-hmac-sha1-96 using AuriStor's proprietary implementation.

New to v0.200 (4 November 2020) - Final release for macOS El Capitan (10.11)

The network path between a client and a server often traverses one or more network segments separated by NAT/PAT devices. If a NAT/PAT times out a RPCs endpoint translation mid-call, this can result in an extended delay before failure and the server being marked down, or worse, a call that never terminates and a client that appears to hang until the fileserver is restarted.

This release includes significant changes to the RX stack and the UNIX cache manager to detect such conditions, fail the calls quickly and detect when it is safe to retry the RPC.

NAT/PAT devices that drop endpoint mappings while in use are anti-social and can result in unwanted delays and even data loss. Whenever possible they should be avoided when they can be. That said, the changes in this release are a huge step to making the loss of endpoint mappings tolerable.

Fix segmentation fault of Backgrounder when krb5_get_credentials() fails due to lack of network connectivity.

Fix the "afsd" rxbind option which was ignored if the default port, 7001, is in use by another process on the system.

If a direct i/o StoreData or FetchData RPC failed such that it must be retried, the retried RPC would fail due to an attempt to Fetch or Store the wrong amount of data. This is fixed.

Servers are no longer marked down if RPCs fail with RX_CALL_PEER_RESET, RX_CALL_EXCEEDS_WINDOW, or RX_PROTOCOL_ERROR. RPCs that are safe to retry are retried.

Fixed a race between a call entering error state and call completion that can result in a the call remaining in the DALLY state and the connection channel remaining in use. If this occurs during process or system shutdown it can result in a deadlock.

During shutdown cancel any pending delayed aborts to prevent a potential deadlock. If a deadlock occurs when unloading a kernel module a reboot will be required.

Updated cellservdb.conf

New to v0.197.1 (31 August 2020) and v0.198 (10 October 2020)

Prevent Dead vnode has core/unlinkedel/flock panic introduced in v0.197.

New to v0.197 (26 August 2020)

A new callback management framework for UNIX cache managers reduces the expense of processing volume callback RPCs from O(number of vcache objects) to O(1). A significant amount of lock contention has been avoided. The new design reduces the risk of the single callback service worker thread blocking. Delays in processing callbacks on a client can adversely impact fileserver performance and other clients in the cell.

Bulk fetch status RPCs are available on macOS for the first time. Bulk fetch status permits optimistic caching of vnode status information without additional round-trips. Individual fetch status RPCs are no longer issued if a bulk status fails to obtain the required status information.

Hardware accelerated crypto is now available for macOS cache managers. AuriStor's proprietary aes256-cts-hmac-sha1-96 and aes256-cts-hmac-sha512-384 implementations leverage Intel processor extensions: AESNI AVX2 AVX SSE41 SSSE3 to achieve the fastest encrypt, decrypt, sign and verify times for RX packets.

This release optimizes the removal of "._" files that are used to store extended attributes by avoiding unnecessary status fetches when the directory entry is going to be removed.

When removing the final directory entry for an in-use vnode, the directory entry must be silly renamed on the fileserver to prevent removal of the backing vnode. The prior implementation risked blindly renaming over an existing silly rename directory entry.

Behavior change! When the vfs performs a lookup on ".", immediately return the current vnode.

if the object is a mount point, do not perform fakestat and attempt to resolve the target volume root vnode.

do not perform any additional access checks on the vnode. If the caller already knows the vnode the access checks were performed earlier. If the access rights have changed, they will be enforced when the vnode is used just as they would have if the lookup of "." was performed within the vfs.

do not perform a fetch status or fetch data rpcs. Again, the same as if the lookup of "." was performed within the vfs.

Volumes mounted at more than one location in the /afs namespace are problematic on more than one operating system that do not expect directories to have more than one parent. It is particularly problematic if a volume is mounted within itself. Starting with this release any attempt to traverse a mountpoint to the volume containing the mountpoint will fail with ENODEV.

When evaluating volume root vnodes, ensure that the vnode's parent is set to the parent directory of the traversed mountpoint and not the mountpoint. Vnodes without a parent can cause spurious ENOENT errors on Mojave and later.

v0.196 was not publicly released.

New to v0.195 (14 May 2020)

This is a CRITICAL update for AuriStorFS macOS clients.

In Sep 2019 AuriStorFS v0.189 was released which provided faster and less CPU intensive writing of (>64GB) large files to /afs. These improvements introduced a hash collision bug in the store data path of the UNIX cache manager which can result in file corruption. If a hash collision occurs between two or more files that are actively being written to via cached I/O (not direct I/O), dirty data can be discarded from the auristorfs cache before it is written to the fileserver creating a file with a range of zeros (a hole) on the fileserver. This hole might not be visible to the application that wrote the data because the lost data was cached by the operating system. This bug has been fixed in v0.195 and it is for this reason that v0.195 has been designated a CRITICAL release for UNIX/Linux clients.

While debugging a Linux SIGBUS issue, it was observed that receipt of an ICMP network error in response to a transmitted packet could result in termination of an unrelated rx call and could mark a server down. If the terminated call is a StoreData RPC, permanent data loss will occur. All Linux clients derived from the IBM AFS code base experience this bug. The v0.195 release prevents this behavior.

This release includes changes that impact all supported UNIX/Linux cache managers. On macOS there is reduced lock contention between kernel threads when the vcache limit has been reached.

The directory name lookup cache (DNLC) implementation was replaced. The new implementation avoids the use of vcache pointers which did not have associated reference counts, and eliminates the invalidation overhead during callback processing. The DNLC now supports arbitrary directory name lengths; the prior implementation only cached entries with names not exceeding 31 characters.

Prevent matching arbitrary cell name prefixes as aliases. For example "/afs/y" should not be an alias for "your-file-system.com". Some shells, for example "zsh", query the filesystem for names as users type. Delays between typed characters result in filesystem lookups. When this occurs in the /afs dynroot directory, this could result in cellname prefix string matches and the dynamic creation of directory entries for those prefixes.

New to v0.194 (2 April 2020)

This is a CRITICAL release for all macOS users. All prior macOS clients whether AuriStorFS or OpenAFS included a bug that could result in data corruption either when reading or writing.

This release also fixes these other issues:

sign and notarize installer plugin "afscell" bundle. The lack of digital signature prevented the installer from prompting for a cellname on some macOS versions.

prevent potential for corruption when caching locally modified directories.

v0.193 was withdrawn due to a newly introduced bug that could result in data corruption.

New to v0.192 (30 January 2020)

The changes improve stability, efficiency, and scalability. Post-0.189 changes exposed race conditions and reference count errors which can lead to a system panic or deadlock. In addition to addressing these deficiencies this release removes bottlenecks that restricted the number of simultaneous vfs operations that could be processed by the AuriStorFS cache manager. The changes in this release have been successfully tested with greater than 400 simultaneous requests sustained for for several days.

New to v0.191 (16 December 2019)

Restore keyed cache manager capability broken in v0.189.

Add kernel module version string to AuriStorFS Preference Pane.

Other kernel module bug fixes.

New to v0.190 (14 November 2019)

Short-circuit busy volume retries after volume or volume location entry is removed.

New to v0.189 (28 October 2019)

Faster "git status" operation on repositories stored in /afs.

Faster and less CPU intensive writing of (>64GB) large files to /afs. Prior to this release writing files larger than 1TB might not complete. With this release store data throughput is consistent regardless of file size. (See "UNIX Cache Manager large file performance improvements" later in this file).

macOS Catalina (8 October 2019)

AuriStorFS v0.188 released for macOS Catalina (10.15)

New to v0.188 (23 June 2019)

Increased clock resolution for timed waits from 1s to 1ns

Added error handling for rx multi rpcs interrupted by signals

New to v0.186 (29 May 2019)

v0.184 moved the /etc/yfs/cmstate.dat file to /var/yfs. With this change afsd would fail to start if /etc/yfs/cmstate.dat exists but contains invalid state information. This is fixed.

v0.184 introduced a potential deadlock during directory processing. This is fixed.

Handle common error table errors obtained outside an afs_Analyze loop. Map VL errors to ENODEV and RX, RXKAD, RXGK errors to ETIMEDOUT

Log all server down and server up events. Transition events from server probes failed to log messages.

RX RPC networking:

If the RPC initiator successfully completes a call without consuming all of the response data fail the call by sending an RX_PROTOCOL_ERROR ABORT to the acceptor and returning a new error, RX_CALL_PREMATURE_END, to the initiator.
Prior to this change failure to consume all of the response data would be silently ignored by the initiator and the acceptor might resend the unconsumed data until any idle timeout expired. The default idle timeout is 60 seconds.

Avoid transmitting ABORT, CHALLENGE, and RESPONSE packets with an uninitialized sequence number. The sequence number is ignored for these packets but set it to zero.

New to v0.184 (26 March 2019)

The initial congestion window has been reduced from 10 Rx packets to 4. Packet reordering and loss has been observed when sending 10 Rx packets via sendmmsg() in a single burst. The lack of udp packet pacing can also increase the likelihood of transmission stalls due to ack clock variation.

The UNIX Cache Manager underwent major revisions to improve the end user experience by revealing more error codes, improving directory cache efficiency, and overall resiliency. The cache manager implementation was redesigned to be more compatible with operating systems such as Linux and macOS that support restartable system calls. With these changes errors such as "Operation not permitted", "No space left on device", "Quota exceeded", and "Interrupted system call" can be reliably reported to applications. Previously such errors might have been converted to "I/O error".

New to v0.180 (9 November 2018)

RX reliability and performance improvements for high latency and/or lossy network paths such as public wide area networks.

A fix for a macOS firewall triggered kernel panic introduced in v0.177.

New to v0.177 (17 October 2018)

A fix to AuriStor's RX implementation bug introduced in v0.176 that interferes with communication with OpenAFS and IBM Location and File Services.

New to v0.176 (3 October 2018)

AuriStor's RX implementation has undergone a major upgrade of its flow control model. Prior implementations were based on TCP Reno Congestion Control as documented in RFC5681; and SACK behavior that was loosely modelled on RFC2018. The new RX state machine implements SACK based loss recovery as documented in RFC6675, with elements of New Reno from RFC5682 on top of TCP-style congestion control elements as documented in RFC5681. The new RX also implements RFC2861 style congestion window validation.

When sending data the RX peer implementing these changes will be more likely to sustain the maximum available throughput while at the same time improving fairness towards competing network data flows. The improved estimation of available pipe capacity permits an increase in the default maximum window size from 60 packets (84.6 KB) to 128 packets (180.5 KB). The larger window size increases the per call theoretical maximum throughput on a 1ms RTT link from 693 mbit/sec to 1478 mbit/sec and on a 30ms RTT link from 23.1 mbit/sec to 49.39 mbit/sec.

Improve shutdown performance by refusing to give up callbacks to known unreachable file servers and apply a shorter timeout period for the rest.

Permit RXAFSCB_WhoAreYou to be successfully executed after an IBM AFS or OpenAFS fileserver unintentionally requests an RX service upgrade from RXAFSCB to RXYFSCB.

RXAFS timestamps are conveyed in unsigned 32-bit integers with a valid range of 1 Jan 1970 (Unix Epoch) through 07 Feb 2106. UNIX kernel timestamps are stored in 32-bit signed integers with a valid range of 13 Dec 1901 through 19 Jan 2038. This discrepency causes RXAFS timestamps within the 2038-2106 range to display as pre-Epoch.

RX Connection lifecycle management was susceptible to a number of race conditions that could result in assertion failures, the lack of a NAT ping connection to each file server, and the potential reuse of RX connections that should have been discarded.

This release includes a redesigned lifecycle that is thread safe, avoids assertions, prevents NAT ping connection loss, and ensures that discarded connections are not reused.

The 0.174 release unintentionally altered the data structure returned to xstat_cm queries. This release restores the correct wire format.

Since v0.171, if a FetchData RPC fails with a VBUSY error and there is only one reachable fileserver hosting the volume, then the VFS request will immediately with an ETIMEDOUT error ("Connection timed out").

v0.176 corrects three bugs that contributed to this failure condition. One was introduced in v0.171, another in 0.162 and the final one dates to IBM AFS 3.5p1.

The intended behavior is that a cache manager, when all volume sites fail an RPC with a VBUSY error, will sleep for up to 15 seconds and then retry the RPC as if the VBUSY error had never been received. If the RPC continues to receive VBUSY errors from all sites after 100 cycles, the request will be failed with EWOULDBLOCK ("Operation would block") and not ETIMEDOUT.

Prefer VOLMISSING and VOLBUSY error states to network error states when generating error codes to return to the VFS layer. This will result in ENODEV ("No such device") errors when all volume sites return VNOVOL or VOFFLINE errors and EWOULDBLOCK ("Operation would block") errors when all volume sites return VBUSY errors. (v0.176)

New to v0.174 (24 September 2018)

macOS Mojave (10.14) support

New to v0.170 (27 April 2018)

Faster processing of cell configuration information by caching service name to port information.

RX call sequence number rollover to permit calls that require the transmission of more than 5.5TB of data.

Command parser Daylight Saving Time bug fix

Fix a bug that prevented immediate access to a mount point created with "fs mkmount" on the same machine.

Fix the setting of "[afsd] sysnames = " during cache manager startup.

New to v0.168 (6 March 2018)

Corrects "fs setacl -negative" processing [CVE-2018-7168]

Improved reliability for keyed cache managers. More persistent key acquisition renewals.

Major refresh to cellservdb.conf contents.

DNS SRV and DNS AFSDB records now take precedence when use_dns = yes

Kerberos realm hinting provided by
kerberos_realm = [REALM]

DNS host names are resolved instead of reliance on hard coded IP addresses
The cache manager now defaults to sparse dynamic root behavior. Only thiscell and those cells that are assigned aliases are included in /afs directory enumeration at startup. Other cells will be dynamically added upon first access.
Several other quality control improvements.

New to v0.167 (7 December 2017)

Addresses a critical remote denial of service vulnerability [CVE-2017-17432]
Alters the volume location information expiration policy to reduce the risk of single points of failures after volume release operations.
'fs setquota' when issued with quota values larger than 2TB will fail against OpenAFS and IBM AFS file servers
Memory management improvements for the memory caches.

New to v0.164 (11 November 2017)

Internal cache manager redesign. No new functionality.

New to v0.160 (21 September 2017)

Support for OSX High Sierra's new Apple File System (APFS). Customers must upgrade to v0.160 or later before upgrading to OSX High Sierra.
Reduced memory requirements for rx listener thread
Avoid triggering a system panic if an AFS local disk cache file is deleted or becomes inaccessible.
Fixes to "fs" command line output

New to v0.159 (7 August 2017)

Improved failover behavior during volume maintenance operations
Corrected a race that could lead the rx listener thread to enter an infinite loop and cease processing incoming packets.

New to v0.157 (12 July 2017)

Bundled with Heimdal 7.4 to address CVE-2017-11103 (Orpheus' Lyre puts Kerberos to sleep!)
"vos" support for volume quotas larger than 2TB.
"fs flushvolume" works
Fixed a bug that can result in a system panic during server capability testing

New to v0.150

AuriStorFS file server detection improvements

New to v0.149

rxkad encryption is enabled by default. Use "fs setcrypt off" to disable encryption when tokens are available.
Fix a bug in atomic operations on Sierra and El Capitan which could adversely impact Rx behavior.

New to v0.128

Extended attribute ._ files are automatically removed when the associated files are unlinked
Throughput improvements when sending data

New to v0.121

OSX Sierra support

New to v0.117

Cache file moved to a persistent location on local disk
AuriStor File System graphics
Improvements in Background token fetch functionality
Fixed a bug introduced in v0.44 that could result in an operating system crash when enumerating AFS directories containing Unicode file names (v0.106)
El Capitan security changes prevented Finder from deleting files and directories. As of v0.106, the AuriStor OSX client implements the required functionality to permit the DesktopHelperService to securely access the AFS cache as the user permitting Finder to delete files and directories.

Features:

Not vulnerable to OPENAFS-SA-2015-007.
Office 2011 can save to /afs.
Office 2016 can now save files to /afs.
OSX Finder and Preview can open executable documents without triggering a "Corrupted File" warning. .AI, .PDF, .TIFF, .JPG, .DOCX, .XLSX, .PPTX, and other structured documents that might contain scripts were impacted.
All file names are now stored to the file server using Unicode UTF-8 Normalization Form C which is compatible with Microsoft Windows.
All file names are converted to Unicode UTF-8 Normalization Form D for processing by OSX applications.

Known issues:

None

macOS Installer (10.11 El Capitan)

Release Notes

Known Issues

If the Kerberos default realm is not configured, a delay of 6m 59s can occur before the AuriStorFS Backgrounder will acquire tokens and display its icon in the macOS menu. This is the result of macOS performing a Bonjour (MDNS) query in an attempt to discover the local realm.
"aklog" will fail to acquire tokens if the current working directory is located in an inaccessible AFS volume. When the "cwd" is inaccessible, the Heimdal framework is unable to initialize the Kerberos v5 context and reports an out of memory error.

New v2021.05-64 (22 Aug 2025)

The v2021.05-64 release fixes a race condition which can lead to a kernel panic when executing the "unlog" command.

v2021.05-63 (31 May 2025)

When AuriStorFS cache managers added support for per-file ACLs in v0.1, the masking of access rights based upon the inode's S_IRWXU mode bits were skipped when reading objects from AuriStorFS fileservers. The masking continued to be applied when reading objects from OpenAFS fileservers. This release applies S_IRWXU mode bits masking of granted access rights to both AuriStorFS and OpenAFS fileservers.
The existing fake stat mountpoint evaluation logic assumes that the fileserver returned a AFSFetchStatus.ParentVnode that refers to the parent directory from which the mountpoint was located. However, this is only guaranteed to be true if the mountpoint does not have a per-file ACL (which is permitted) and has never been cross-directory hard linked (which is also permitted).
This release improves the error handling logic to fail the fake stat evaluation if the AFSFetchStatus.ParentVnode does not match expectations.
This release fixes a metadata inconsistency when a mountpoint is created when disconnected mode is enabled. A mountpoint should be reported to the vfs as a directory and not a symlink.
The cache manager must track for each inode whether the object's status infomration was received from an AFS3 fileserver or an AuriStorFS fileserver. This release alters when that information is recorded to ensure that it is recorded each time the inode's status information is updated by a fileserver response.
This knowledge is required to properly determine how to interpret the status information.
Directory entries referring to mountpoints and volume roots must not be stored within the Directory Name Lookup Cache (DNLC). Instead, full lookups must be performed each time to ensure proper mountpoint resolution.
Prior to this release locally created mountpoints were stored into the DNLC by mistake.

v2021.05-62 (24 April 2025)

macOS Sequoia v15.4 vs Cisco AnyConnect VPN

Beginning with macOS 15.4, use of Cisco AnyConnect VPN breaks DNS SRV, AFSDB and TXT record lookups performed using the cross-platform DNS resolver API. As a result, clients which rely upon DNS-based zero-configuration resolution of Kerberos realms and/or AFS cells database services are unable to access those resources. This release switches the macOS builds to use the Apple DNS API.
The lookup failures impacted all command line tools, clients and services.
Note: Communication failures can occur if the VPN is configured by the organization to block the transmission of fragmented UDP packets. Set [afsd] rxmaxmtu = 1328 to ensure all RX packets are small enough to traverse the VPN link.
This release improves consistency of vnode metadata interpretation when the status information is received via an RPC which does not return a callback.

v2021.05-61 (31 March 2025)

Instead of allocating the same sized hash table for the vcache and dcache objects regardless of how many elements are expected to be stored, scale the hash table sizes. The vcache hash table will scale from 4096 buckets to 131072 buckets with a target of 4 entries per bucket. The dcache hash tables will scale from 4096 buckets to 65536 buckets with a target of 2 entries per bucket.
Historically, afs cache managers which create symlinks (including mountpoints) locally allocate a fake infinite lifetime callback to permit applications to use the symlink (or mountpoint) without requiring FetchStatus RPCs to be issued to the fileserver. This approach has drawbacks because it means that fileservers will not issue a callback break when the volume containing the symlink is taken offline, moved, deleted, etc. It also means that a callback will not be sent if another cache manager removes the symlink.
Beginning with this release the AuriStorFS cache manager will no longer allocate a fake infinite lifetime callback and will instead rely upon fileserver allocated callbacks.
"fs makemount" is supposed to create a mountpoint even if the target volume does not exist or is inaccessible. v2021.05 introduced support for creating mountpoints which kafs which broke the ability to create mountpoints to non-existent volumes.

v2021.05-60 (6 March 2025)

The cache manager retries fileserver RPCs up to 100 times when the fileserver replies with VBUSY, VOFFLINE or VSALVAGE. The cache manager fails the application request after the limit is reached. Prior to this release, the failure would result in the cache manager returning an ENODEV error to the application. But ENODEV indicates that the required volume does not exist and the fileserver has not indicated that the volume does not exist. The cache manager will now return EWOULDBLOCK to the application because the volume might be temporarily inaccessible.
Many OpenAFS and AuriStorFS fileservers can return VNOVOL as a side effect of a race between a volume going offline so that control can be given to the Volume Management Service (volserver) and an incoming RPC requesting that the volume be attached. A VNOVOL can also be returned because the volume was temporarily deleted from the vice partition prior to being replaced by a new clone, or prior to the fileserver marking the volume group as having been moved to another fileserver.
The fileserver replying to an RPC with VNOVOL cannot be trusted to mean that the volume has been permanently removed unless it is matched by the Location Service reporting the Volume Location (VL) entry has been deleted. From now on when a VNOVOL response is received and the VL entry exists, the cache manager will return EWOULDBLOCK instead of ENODEV to the application.
Prior to this release, any VL error code returned from the Location Service would be interpreted as meaning that the requested VL entry does not exist. Starting with this release only the VL_NOENT error will be interpreted as the VL entry does not exist.
Restrict the internal use of the ENOENT error code so the only use of ENOENT is to indicate that a directory entry does not exist. Previously, ENOENT could be returned if a disk cache entry could not be allocated or found. A disk cache related ENOENT returned when processing a directory might have been confused for the requested directory entry not existing.
Restrict the internal use of the ENODEV error code so that the only meaning of ENODEV is that the required volume does not exist. Previously, ENODEV could be returned due to transient errors when failing to contact the Location Service.

v2021.05-59 (13 February 2025)

The fix for garbage collection of idle unixuser objects introduced in v2021.05-56 was incomplete. As a result, Rx client connections might not be garbage collected.

v2021.05-57 (9 February 2025)

No new changes for macOS.

v2021.05-56 (7 February 2025)

Minor bug fixes to the Rx RPC implementation
The fix for garbage collection of idle unixuser objects introduced in v2021.05-55 was incomplete. As a result, Rx client connections might not be garbage collected.

v2021.05-55 (23 January 2025)

Minor bug fixes to the Rx RPC implementation
During garbage collection of idle unixuser objects, any Rx client connections associated with the unixuser (uid or pag) that were in use by another thread would not have their cache manager reference dropped. As a result the Rx client connections could never be garbage collected which is problematic on systems which churn through many unixusers. The behavior has been changed so that unixusers objects with in use Rx connections will not be freed. This permits future attempts to garbage collect unixusers to do so only after the Rx connections can also be freed.
Although this misbehavior has been present for decades, it is only since v2021.05-52 that the problem has been observed. It is possible that the changes to reduce the holding of the GLOCK has created a window of opportunity for connections to be simultaneously in use during unixuser garbage collection.

v2021.05-54 (19 January 2025)

The UNIX cache manager changes continue AuriStor's efforts to address a report that was received at the end of September. The end user site uses an OpenAFS cell to distribute software for execution by IBM Spectrum LSF (load sharing facility) managed compute nodes. Each of the approximately one dozen compute nodes is configured with approximately 100 processor cores and 1TB of ram. Each job scheduled by LSF runs in a newly allocated Process Authentication Group (PAG). Some jobs run for hours or days; others complete in a few seconds. LSF has been observed scheduling more than 3000 jobs in under a minute with peak counts of cache manager user objects at 1300. Everything ran well until it didn't. After some unexplained trigger event, all of the remaining free memory on the system would be consumed in about an hour causing the system to panic.

Over the last five months AuriStor has diligently worked to develop theories, optimize the AuriStorFS UNIX cache manager to address each theory; then move on to subsequent ideas once prior theories were disproved. Over this time we have replaced the memory management lifecycle for access rights entries and rx events. We have made it possible for Linux permission checks to be processed without acquiring the GLOCK in the majority of access patterns thereby increasing parallelism. The lifecycle management of PAG allocation and user entity tracking has been rewritten to avoid the GLOCK. The Linux inode lifecycle management has been rewritten which some sites attribute to elimination of swapping on their systems. All of these changes increased throughput and parallelism within and without the cache manager. However, none of them prevented the memory consumption once the unknown trigger event occurred.

It is with great pleasure that we can report that the trigger event and the source of the memory consumption have been identified. The trigger is a damaged Rx packet header which causes the AuriStor Rx stack to place an Rx connection into an RX_PROTOCOL_ERROR error state when there is an in-flight call to an OpenAFS fileserver. The memory consumption is due to the infinite loop that occurs when an Rx connection is in an error state and the error code is not one of the error codes which forces replacement of the Rx connection. Within each pass of the loop a structure is allocated to track the history of per-server errors so they can be logged when hard mount mode is enabled for the cell. AuriStor Rx permits up to 12 calls on each Rx connection. As a result there can be up to a dozen processes each spinning and allocating memory when the Rx connection is placed into the error state.

Days before Christmas 2023 AuriStor identified a race condition which can result in damaged Rx DATA packets being written to the network. Initial transmission of Rx DATA packets can be sent either from an application thread or an Rx listener. Packet retransmission occurs either in an Rx listener thread or in the Rx event thread. The race permits a second thread to modify the Rx packet headers while the sendmsg() syscall from the original transmission is executing. This race is possible because the original thread might exhaust its timeslice during the sendmsg() call and it might be many milliseconds before it is rescheduled on a busy system. In the meantime, packet retransmissions might be initiated either by an Rx listener thread processing an ACK packet or the Rx event thread processing a timeout event. If the initial packet transmission completes while the Rx headers are being rewritten the resulting packet sent on the wire might be garbage. AuriStor observed that if the modified packets were jumbograms that silent data corruption could occur so we disabled all use of jumbograms in v2021.05-35 We also notified both IBM and OpenAFS of the problem.

v2021.05-53 modified the RPC result analyze logic to force the replacement of Rx connections in an error state except when the error signals a token expiration. When a token is expired there is no benefit to replacing the connection until new tokens have been acquired. This release further ensures that infinite memory allocation cannot occur by tracking only the most recent error per fileserver; and only does so if hard mount mode is enabled for the cell.

Processes that issued syscalls which became stuck in a RPC retry loop could not be interrupted. In addition to the unintentional retry loops there are intentional retry loops such as those that occur because a fileserver repeated fails the call with a “VBUSY” error. “VBUSY” originally meant that the required volume was temporarily in use by the volume management service (“volserver”) and would be back in service soon so please wait. Beginning with IBM AFS 3.4 the error code was used to force a retry for reasons having nothing to do with the requested volume’s availability. IBM AFS 3.4 introduced the concept of multi-homed clients which meant that the fileserver could no longer track the identity of a cache manager by its IPv4 address. Instead, each cache manager randomly generates a UUID which the fileserver must query each time an incoming RPC is received on a previously unrecognized Rx connection. An Rx connection might be unrecognized because it’s new, or because the client’s IP address changed, or because the fileserver garbage collected its state information. Regardless of the reason, if a call is received and there is no Rx connection binding to a cache manager UUID, then the fileserver application worker thread to which the call was assigned must issue an outgoing RPC to the client’s RXAFSCB service. Each client has at most one worker thread. Prior to this release, every incoming RXAFSCB call required acquisition of the GLOCK before it could execute. If the GLOCK is unavailable, the RXAFSCB thread blocks and no other pending RXAFSCB calls can be processed. If a RXAFSCB call blocks for too long, the fileserver timeouts trigger and the fileserver replies to the incoming RXAFS call with “VBUSY”. “VBUSY” can also be sent if the fileserver’s quota on the number of calls from the same endpoint is exceeded.

The v2021.05-54 release removes the GLOCK requirement for many RXAFSCB RPCs including all of the UUID query RPCs and all whole volume callback RPCs. Callback RPCs for individual vnodes must still acquire the GLOCK. However, it is hoped that these changes will significantly reduce the time required to process the queries generated due to unrecognized Rx connections. The AuriStor Rx stack permits a dozen calls to be issued on a single logical connection which the fileserver sees as three unique Rx connections from the same endpoint.

Each Unix UID or PAG ID must use an independent Rx connection to each fileserver. On a system which processes /afs syscalls for 1200 UID or PAG IDs in a minute, there is the potential for 3600 Rx connections per fileserver. When the fileserver is OpenAFS, this quickly triggers per endpoint throttling and generates VBUSY storms which can result in unlucky client processes experiencing starvation. With this release, starving processes can be interrupted and RXAFSCB UUID queries can be completed without blocking.

AuriStor believes this release will provide significant benefits for large scale compute nodes, web servers and time sharing systems.

v2021.05-53 (4 January 2025)

v2021.05-52 switched the memory allocation strategy for rx_event objects from a global free list to reliance upon the operating system's memory allocator to allocate individual objects which are freed upon rx_event completion or cancelation. Reliance upon the system allocator introduces the risk of allocation failures due to out of memory conditions. If allocation of an rx_event object fails, a reference count leak on the associated rx_connection or rx_call object can occur preventing garbage collection of the object.
This release handles rx_event allocation failures:
- delayed ACK or delayed ABORT events are replaced by immediate transmission of the ACK or ABORT
- other cases fail the call or connection with an RX_OUT_OF_MEMORY error which is translated on the wire to RX_WIRE_CALL_DEAD (-1).
After executing a call to a fileserver, if the Rx connection is in an error state for any reason other than expired tokens, then schedule the Rx connection to be replaced the next time a call is attempted. All calls issued on an Rx connection that is already in an error state fail immediately preventing further communication with the associated fileserver from the UID/PAG.
Rx connections in an expired token error state are not replaced until new tokens are set.
Rx connections can be placed into an error state either due to local or remote failures. Such failures are often generated by the in-use security class (RXKAD/RXGK) but might be due to an out-of-memory condition, system clock advancement or reversal, or even application errors.
This release includes internal code changes to the management of servers, connections and access rights which will be used by a future release to separate PAG ID allocation from the system's UID namespace.
If 'afsd' is configured with a keytab, receipt of a PoweredOn event will force an attempt to acquire system tokens.
On supported macOS versions, enforce the use of code signing identities to authenticate mach messages involving the AFSBackgrounder and the AFSPreference panel.

v2021.05-52 (5 December 2024)

The v2021.05-50 replacement of the linked list implementation of the vcache access cache with a btree and last lookup cache introduced a memory corruption bug which is fixed in this release.
Eliminate the GLOCK requirement for the collection of function call and RPC statistics collection.
Replace small space allocation with a GLOCK free allocator.
Reimplement the cache manager tracing logging routines so they no longer require the GLOCK.
The Rx networking stack schedules many different types of time sensitive events associated with individual packet retransmission, delayed ack and abort processing, path mtu detection, keep alives, and other categories. Prior to this release, all memory allocated to Rx events objects would not be freed to the operating system until the userspace process terminated or the kernel module was unloaded.
Beginning with this release, instead of having a single, global, free list for rxevents, the system allocator is trusted to be able to allocate memory quickly. This simplifies the Rx implementation, and should be more efficient, as most system allocators will have per thread or core slabs, removing the contention of the global lock protecting the rxevent free list. In addition, the change protects against a network event allocating memory that cannot be readily freed.
On non-Total Store Ordering (TSO)architectures, relaxed atomics used as reference counters do not ensure completion of all prior stores or recency of later loads. This release replaces all read-modify-write atomics to be acquire-release, which, among other things, ensures that only the intended ordering is possible.

v2021.05-51 (26 November 2024)

Fix a vcache (aka inode) reference leak that can occur during an access rights query for a non-directory object if the neither the calling process nor the cache manager possesses tokens for the cell containing the object. The leak was introduced in v2021.05-45.
When scanning the token list for each user searching for expired tokens the user spinlock must be held to prevent races. The updated user management implementation pulled into v2021.05-45 failed to hold the spinlock.
During cache manager module shutdown reset global structures to the module initialization state. Doing so improves the likelihood that the module can be reinitialized.

v2021.05-50 (23 November 2024)

The linked list implementation of the vcache access rights cache was replaced with a btree and a last lookup cache. The btree design is more efficient for lookups and removals, and scales better as the number of uids and/or process authentication groups (pags) accessing /afs increases.
Prune access rights for expired users. Doing so reduces the cost of subsequent lookups.
Cache the access rights obtained for a uid|pag via the "anyAccess" field. Previously, a uid|pag that obtained its rights from the "anyAccess" field had to perform and fail a rights lookup on each and every vcache access.
When setting or clearing tokens for a single cell, no longer invalidate the access rights cached for the uid|pag for all cells. Instead, only invalidate the access rights cached for the cell whose tokens were set or cleared.

v2021.05-49 (16 November 2024)

The output of "tokens" command failed to report yfs-rxgk tokens was broken starting in v2021.05-46

v2021.05-48 (12 November 2024)

Preallocated buffer overflows in XDR responses (CVE-2024-10397)
The AuriStorFS and AFS3 RPC suites rely upon Sun RPC XDR to marshal binary data structures for network transfer. The AuriStor XDR implementation is derived from Sun Microsystems' Sun RPC code base. The Sun RPC XDR API permits memory for output parameters to (optionally) be preallocated which can result in various classes of memory corruption and/or memory leaks in RPC initiator processes.
The AuriStorFS v2021.05-48 release introduces additional data length validation checks within the AuriStor XDR implementation and prohibits the use of preallocated memory for string output parameters or fields. All cache managers, servers and command line tools are modified by these changes.

v2021.05-46 (28 October 2024)

Cache Manager:
- Prevent a kernel memory leak when server preferences are set via the yfs-client.conf [afsd] configuration or via "fs setserverprefs".
- Directory enumeration of a truncated directory now returns an error instead of assuming the end of the directory has been reached.
- Since AFS 3.0, the Unix cache manager has used the root identity credentials to create anonymous outgoing connections to the location service and each fileserver. However, if uid 0 is assigned a token, then those Rx connections will no longer be anonymous. Beginning with this release anonymous outgoing connections are always created with the NOPAG identity (uid 0xffffffff) instead of the root identity.
- When establishing an outgoing rxgk connection, do not fallback to the systemuser's credentials if the user's credentials resulted in a fatal error. Falling back to the systemuser's credentials can result in inappropriate use of an anonymous connection.
- Improved access rights cache correctness for YFS servers
  In prior releases, the access check logic used the file rights for any files fetched from an AuriStorFS fileserver. For files fetched from an AFS-3 fileserver (and, historically, for all files), it used the directory rights, with the (a)dmin right from the file mixed in. The (a)dmin right on a non-directory indicates that the object is owned by the authenticated user.
  This approach has some issues when combined with the access rights cache, and current fileserver callback behavior. On an AuriStorFS file server, the rights on a non-directory may be determined by the rights granted on its parent directory or, with per-file ACLs, those granted on the object itself. The fileserver will only break a non-directory's callback when a per-file ACL is changed - changing a directory ACL will not break callbacks on files within that directory. This means that changing a directory ACL will not invalidate access rights cache entries on files in tha directory, even if the effective ACL on this files has changed, and the cached rights are no longer correct.
  This release works around this by adding a new function which returns the access rights for a file hosted on an AuriStor fileserver. It uses the parent vnode information to locate the parent directory. If the parent directory isn't in the cache, or it doesn't have a valid callback, or if it has been changed since the file's access rights were cached, it clears the current access rights. Files without a parent directory must have per-file ACLs, and so their cached rights can be safely used.
  Note that files with parent vnodes may still have per-file ACLs, and that the breadcrumbing performed by the client may add parent vnode fields to vnodes which don't have them provided by the fileserver. Such vnodes may have their cached access rights cleared more frequently than necessary.
- Add a new mechanism for caching access rights within the vcache structure. This cache is protected via a vcache specific spinlock, and can be accessed without holding the GLOCK.
  This new cache mechanism returns the memory associated with cached rights back to the kernel's slab free memory pool instead of adding the unused rights structures to a cache manager managed free list. The previous cache implementation never returned allocated memory to the kernel. Instead, invalidated access rights were appended to a free access rights queue for later reuse.
- When a volume is accessed via multiple mountpoints a choice must be made regarding which mountpoint is considered to be the active (or parent) mountpoint. This release alters the behavior such that the active mountpoint is set every time a mountpoint is traversed.
  This behavior is easier to understand and is more likely to provide the expected result for a single process that repeatedly accesses volumes from multiple mountpoints. However, it can result in unexpected results when multiple processes are traversing multiple mountpoints in parallel without any synchronization.

v2021.05-45 (Not released)

v2021.05-44a (18 September 2024)

Authentication:
- AuriStorFS v2021.05-44 included an updated version of the Heimdal Kerberos framework used by AuriStorFS when acquiring yfs-rxgk and rxkad authentication tokens. The updated Heimdal included a bug which disabled the use of DNS SRV records for KDC discovery and DNS TXT records for realm discovery. As a side effect token acquisition might fail with an unable to reach any KDC in realm error. This is fixed in v2021.05-44a.

v2021.05-44 (17 August 2024)

Cache Manager:
- Since v0.192 the cache manager has failed acquire the global lock when upgrading a shared-lock to a write-lock during the execution of a background cache chunk file truncation.
Authentication:
- Neither MIT nor Heimdal gssapi nor their gss mechanisms consistently initialize the output 'minorStatus' parameter. Various functions can return either success or failure majorStatus values with minorStatus unassigned. As a result, stack garbage will be used when generating error messages. From now on libyfs_acquire will always initialize the minorStatus output variable to zero before calling into the gssapi library.
Command Parser:
- No longer accept the token "-" as a switch which eventually fails with a CMD_UNKNOWNSWITCH error. Instead, process the token as a data value.
- Optimize the processing of the loop which processes "source" command input.
- If the source command input file is "-", read from stdin.

v2021.05-41 (26 June 2024)

Rx Networking (libyfs_rx):
- A race during event creation can lead to the freeing of the event while its still in use.
- RFC1122 says that Net and Host unreachable ICMP errors might be transient and should therefore not be treated as fatal. There is no such language for the equivalent ICMPV6 errors. However, in practice ICMP6_DST_UNREACH_NOROUTE, ICMP6_DST_UNREACH_BEYONDSCOPE, and ICMP6_DST_UNREACH_ADDR can be transient.
  Linux has considered these ICMPV6 destination unreachable errors as non-fatal going back at least as far as the initial git repository commit.
  AuriStor Rx has always treated these as fatal errors which results in immediate termination of in-flight calls when received. Even if the network route corrects itself before the call timeout period expires. This release mirrors the Linux behavior and makes these errors non-fatal.
Cache Manager:
- For the first time the cache manager can detect the deletion of a volume and handle the creation of a new volume with the same name but a different volume id.
- If the location service reports the deletion of a volume, invalidate all mount points to that volume.
- RXAFS_GetCapabilities RPC failures should not be treated as a fatal error preventing failover to another replica site.
Authentication ("libyfs_acquire") used by aklog, vos, pts, bos, afsio:
- rxkad_k5 token acquisition krb5 ccache management. This release altered the krb5 credential cache management strategy once again to work around different bugs in MIT krb5 and Heimdal.
- New ACQUIRE_ERR_CRED_EXPIRED error code introduced to represent the case when a request for a service credential returns one that is already expired.
Command parser (libyfs_cmd):
- When parsing configuration files there is a depth limit of ten active inclusions. This limit was improperly enforced as a limit of ten included files instead of a depth of ten included files. As of this release it is now possible to populate an includedir directory with any number of .conf files.

v2021.05-40

Not released.

v2021.05-39 (20 May 2024)

Parallel Random Number Generation:
- AuriStorFS processes rely upon the krb5_generate_random() and RAND_bytes() functions to obtain random bytes for cryptographic operations and random counters. krb5_generate_random() internally acquires a mutex to protect internal state information. This mutex has become a significant barrier to the encryption and checksumming of Rx packets with both yfs-rxgk and rxkad.
  
  This release replaces general use of krb5_generate_random() and RAND_bytes() with a per-thread ChaCha20 CS-PRNG. This avoids the acquisition of a global mutex and permits increased parallelism on multi-core systems.
Rx Networking (libyfs_rx):
- The Rx network stack schedules a garbage collection operation to execute once per minute. This operation enforces call timeouts, destroys idle connections and destroys idle peers. The operation has historically been performed by the Rx event thread which is already responsible for performing actions in response to call RTOs, sending NAT Ping and keep-alive packets, and retrying connection challenge and reachability checks.
  
  The time complexity of the garbage collection operation is determined by the number of calls, connections, and peers. The busier the Rx endpoint the more work must be performed during each garbage collection run and the longer it takes to complete. While garbage collection is active other events cannot be processed which can interfere with the proper flow control of active calls.
  
  As with all Rx events, the garbage collection event is scheduled to execute at an absolute clock time. If the system clock drifts (or is administratively set) backwards garbage collection will not be performed until the clock catches up with the scheduled time.
  
  Another responsibility of the garbage collection procedure is to terminate calls if the system clock drifted backwards by five minutes or longer. However, when the clocked drifts backwards garbage collection is not performed until the clock has advanced beyond the point where calls require termination. As a result, calls are not terminated due to backwards clock drift and they can stall.
  
  This release re-implements the garbage collection procedure using a dedicated thread and relative waits. This change ensures that the garbage collection procedure will not prevent the execution of call related events and permits calls to be terminated when large backward clock drifts are detected.
Disk Cache Management:
- Since IBM AFS 3.5, the cache has been considered "too full" even if there exist cache files have been discarded but not yet truncated. When the cache is "too full" most operations that write to the cache will block until truncation of discarded cache files has been performed which results in unnecessary delays. This release fixes the cache such that that discarded but not yet truncated cache files do not block write operations.
- This release permits the cache truncation daemon thread to exit sooner if the cache manager is shutting down.
Improved failover when the RXGK service (co-located with each vlserver) fails to issue tokens. The failures might be the result of misconfiguration, an inability to read keys or loss of Ubik quorum.

v2021.05-38 (29 February 2024)

As with other AuriStorFS releases since the beginning of 2024, this release includes additional improvements to the Rx RPC implementation which are related to the possibility of silent data corruption when Rx jumbograms are in use. Prior releases disabled the negotiation of Rx jumbograms such that the v2021.05-37 Rx peer will refuse to send Rx jumbograms and will request that the remote peer does not send them. However, a bad actor could choose to send Rx jumbograms even though they were asked not to. v2021.05-38 introduces additional protections to ensure that a corrupt Rx jumbogram is dropped instead of being accepted.

The v2021.05-38 Rx RPC implementation also includes two optimizations. First, when Rx initiators complete a call they will no longer send an extra ACK packet to the Rx acceptor of the completed call. The sending of this unnecessary ACK creates additional work for the server which can result in increased latency for other calls being processed by the server.

Second, all AuriStor Rx services require a reach check for incoming calls from Rx peers to help protect against Distributed Reflection Denial of Service (DRDoS) attacks and execution of RPCs when the response cannot be delivered to the caller. A new reach check is required for each new call that arrives more than 60 seconds after the prior reach check completed. v2021.05-38 Rx considers the successful acknowledgment of a response DATA packet as a reach check validation. With this change reach checks will not be periodically required for a peer that completes at least one call per 60 seconds. A 1 RTT delay is therefore avoided each time a reach check can be avoided. In addition, reach checks require the service to process an additional ACK packet. Eliminating a large number of reach checks can improve overall service performance.

The final Rx RPC change in this release is specific to kernel implementations. Prior releases restricted the frequency of executing time scheduled Rx events to a granularity no smaller than 500ms. As a result an RTO timer event for a lost packet could not be shorter than 500ms even if the measured RTT for the connection is significantly smaller. The minimum RTO for a connection in AuriStor Rx is 200ms. The inability to schedule shorter timeouts impacts recovery from packet loss.

v2021.05-37 (5 February 2024)

Rx improvements:
- The v2021.05-36 release permanently disabled all use of Rx jumbograms due to a risk of silent data corruption. However, when advertising the number of acceptable datagrams in the ACK trailer a missing htonl() set the value to 16777216 instead of 1 on little-endian systems.
- When sending a PING ACK as a reachability test, ensure that the previousPacket field is properly assigned to the largest accepted DATA packet sequence number instead of zero.
- Replace the initialization state flag with two flags. One that indicates that Rx initialization began and the other that it succeeded. The first prevents multiple attempts at initialization after failure. The second prevents shutdown from accessing uninitialized structures if initialization failed.
Cache Manager Improvements:
- No longer refuse to start if both the 'cachedir' and 'memcache' options are present in the configuration file.
- New variable to store the maximum number of cache blocks used. which is accessible via /proc/fs/auristorfs/cache/blocks_used_max.

v2021.05-36 (10 January 2024)

Rx improvements:
- Permanently disable all use of Rx jumbograms due to a risk of silent data corruption.
  
  Ever since OpenAFS 1.0, and possibly before, a race condition has existed when Rx transmits packets. As the rx_call.lock is dropped when starting packet transmission, there is no protection for data that is being copied into the kernel by sendmsg(). It is critical that this packet data is not modified by another thread. However, races exist between the application, listener, and event threads that can lead to retransmissions starting whilst an original transmission is still in progress. This can lead to the packet headers being overwritten, and either the original transmission, the retransmission or both sending corrupt data to the peer.
  
  This corruption can affect the packet serial number or packet flags. It is particularly harmful when the packet flags are corrupted, as this can lead to multiple Rx packets which were intended to be sent as Rx jumbograms being delivered and misinterpreted as a single large packet. The eventual result of this depends on the Rx security class in play, but it can cause decrypt integrity errors (rxgk:crypt and rxgk:auth) or corruption of the data stream (rxnull, rxgk:clear or rxkad:auth).
  
  All AuriStorFS servers, OpenAFS 1.6 or later servers, and the Windows cache manager have been shipped with Rx jumbograms disabled by default. The UNIX cache managers however are shipped with jumbograms enabled. There are many AFS cells around the world that continue to deploy OpenAFS 1.4 or earlier fileservers which continue to negotiate the use of Rx jumbograms.
  
  It is worth noting that all AuriStorFS v0.198 and later fileservers and cache managers implement explicit checks that will recognize the corrupted application data stream and prevent corrupted file content from being stored either into an AFS vnode's backing store in the volume's object store or the cache manager's AFS cache. OpenAFS cache managers and fileservers do not have these protections.
  
  With Rx jumbograms disabled the maximum number of Rx packets in a datagram is reduced from 6 to 1; the maximum number of send and receive datagram fragments is reduced from 4 to 1; and the maximum advertised MTU is restricted to 1444 - the maximum rx packet size prior to the introduction of jumbograms in IBM AFS 3.5.
- If the rx call flow state transitions from either the RECOVERY or RESCUE states to the LOSS state as a result of an RTO resend event while writing packets to the network, cease transmission of any new DATA packets if there are packets in the resend queue.
- When the call flow state is LOSS and all packets in the resend queue have been retransmitted and yet the recovery point has not been reached, then permit new DATA packets to be sent in order to maintain a full congestion window.
- Add a safety check to prevent the estimated RTT from underflowing when the actual roundtrip time is smaller than 125us.
- Fix the computation of the padding required for rxgk encrypted packets. This bug results in packets sending 8 bytes fewer per packets than the network permits. This bug accidentally prevented the construction of Rx jumbograms when a call is protected by rxgk:crypt.
- Replace the random number generator with a more security source of random bytes.

v2021.05-33 (27 November 2023)

Rx improvements:
- Not all calls transfer enough data to be able to measure a smoothed round-trip time (SRTT). Calls which are unable to compute a SRTT should not be used to update the peer host RTO value which is used to initialize the RTO for subsequent calls.
  
  Without this change, a single DATA packet call will cause the peer host RTO to be reduced to 0ms. Subsequent calls will start with a RTO value of MAX(0, rxi_minPeerTimeout) where rxi_minPeerTimeout defaults to 200ms. If the actual measured RTO is greater than 200ms, then initial RTO will be too small resulting in premature triggering of the RTO timer and the call flow state entering the loss phase which can significantly hurt performance.
- Initialize the peer host RTO to rxi_minPeerTimeout (which defaults to 200ms) instead of one second. Although RFC6298 recommends the use of one second when no SRTT is available, Rx has long used the rxi_minPeerTimeout value for other purposes which are supposed to be consistent with initial RTO value. It should be noted that Linux TCP uses 200ms instead of one second for this purpose.
- If associating a security class with an Rx connection fails immediately place the Rx connection into an error state. A failure might occur if the security class is unable to access valid key material.
- If an incoming Rx call requires authentication and the security class is unable to successfully generate a challenge, put the incoming Rx connection into an error state and issue an abort to the caller.
- If an incoming Rx call requires authentication and the security class is able to generate a challenge but the challenge cannot be returned to Rx, then treat this as a transient error. Do not acknowledge the incoming DATA packet and do not place the Rx connection into an error state. An attempt to re-issue the challenge will be performed when the DATA packet is retransmitted.
- If an Rx call is terminated due to the expiration of the configured connection dead time, idle dead time, hard dead time, or as a result of clock drift, then send an ABORT to the peer notifying them that the call has been terminated. This is particularly important for terminated outgoing calls. If the peer does not know to terminate the call, then the call channel might be in use when the next outgoing call is issued using the same call channel. If the next incoming call is received by an in-use call channel, the receiver must drop the received DATA packet and return a BUSY packet. The call initiator will need to wait for a retransmission timeout to pass before retransmitting the DATA packet. Receipt of BUSY packets cannot be used to keep a call alive and therefore the requested call is at greater risk of timing out if the network path is congested.
aklog and krb5.log (via libyfs_acquire):
- If the linked Kerberos library implements krb5_cc_cache_match() and libacquire has been told to use an explicit principal name and credential cache, the Kerberos library might return KRB5_CC_NOTFOUND even though the requested credential cache is the correct one to use. This release will not call krb5_cc_cache_match() if the requested credential cache contains the requested principal.
Cell Service Database (cellservdb.conf):
- cellservdb.conf has been synchronized with the 31 Oct 2023 update to the grand.central.org CellServDB file.

v2021.05-32 (9 October 2023)

No significant changes for macOS compared to v2021.05-31

v2021.05-31 (25 September 2023)

New platform:
- macOS 14 Sonoma
macOS 14 Sonoma:

AuriStorFS v2021.05-29 and later installers for macOS 13 Ventura are compatible with macOS 14 Sonoma and do not need to be removed before upgrading to macOS 14 Sonoma. Installation of the macOS 14 Sonoma version of AuriStorFS is recommended.

Cache Manager:
- If an AuriStorFS cache manager is unable to use the yfs-rxgk security class when communicating with an AuriStorFS fileserver, it must assume it is IBM AFS 3.6 or OpenAFS and upgrade it to AuriStorFS if an upgrade probe returns a positive result. Once a fileserver's type is identified as AuriStorFS the type should never be reset; even if communication with the fileserver is lost or the fileserver restarts.
  
  If an AuriStorFS fileserver is replaced by an OpenAFS fileserver on the same endpoint, then the UUID of the OpenAFS must be different. As a result, the OpenAFS fileserver will be observed as distinct from the AuriStorFS fileserver that previously shared the endpoint.
  
  Prior to this release there were circumstances in which the cache manager discarded the fileserver type information and would fail to recognize the fileserver as an AuriStorFS fileserver when yfs-rxgk could not be used. This release prevents the cache manager from resetting the type information if the fileserver is marked down.
- If a fileserver's location service entry is updated with a new uniquifier value (aka version number), this indicates that one of the following might have changed:
  1. the fileserver's capabilities
  2. the fileserver's security policy
  3. the fileserver's knowledge of the cell-wide yfs-rxgk key
  4. the fileserver's endpoints
  Beginning with this release the cache manager will force the establishment of new Rx connections to the fileserver when the uniquifier changes. This ensures that the cache manager will attempt to fetch new per-fileserver yfs-rxgk tokens from the cell's RXGK service, enforce the latest security policy, and not end up in a situation where its existing tokens cannot be used to communicate with the fileserver.
aklog:
- Fix incorrect output when populating the server list for a service fails. The stashed extended error explaining the cause of the failure was not displayed.
- If a cell has neither _afs3-prserver._udp. DNS SRV records nor AFSDB records, the lookup of the cell's protection servers would fail if there is no local cell configuration details. The fallback to use _afs3-vlserver._udp. DNS SRV records did not work. This is corrected in this release.

v2021.05-30 (6 September 2023)

Do not mark a fileserver down in response to a KRB5 error code.
fs cleanacl must not store back to the file server a cleaned acl if it was inherited from a directory. Doing so will create a file acl.
Correct the generation of never expire rxkad_krb5 tokens from Kerberos v5 tickets which must have a start time of Unix epoch and an end time of 0xFFFFFFFF seconds. The incorrectly generated tokens were subject to the maximum lifetime of 30 days.
Correct the generation of the yfs-rxgk RESPONSE packet header which failed to specify the key version generation number used to encrypt the authenticator. If the actual key version is greater than zero, then the authenticator would fail to verify.
Enforce a maximum NAT ping period of 20s to ensure that NAT/PAT/firewall rules due not expire while Rx RPCs are in-flight.

v2021.05-29 (26 June 2023)

Execution of fs commands such as examine, whereis, listquota, fetchacl, cleanacl, storeacl, whoami, lsmount, bypassthreshold and getserverprefs could result in memory leaks by the AuriStorFS kernel extension.

v2021.05-27 (1 May 2023)

Fixes for bugs in vos introduced in v2021.05-26.

v2021.05-26 (17 April 2023)

Fixed a potential kernel memory leak when triggered by fs examine, fs listquota, or fs quota.
Increased logging of VBUSY, VOFFLINE, VSALVAGE, and RX_RESTARTING error responses. A log message is now generated whenever a task begins to wait as a result of one of these error responses from a fileserver. Previously, a message was only logged if the volume location information was expired or discarded.
Several changes to optimize internal volume lookups.
Faster failover to replica sites when a fileserver returns RX_RESTARTING, VNOVOL or VMOVED.
rxdebug regains the ability to report rx call flags and rx_connection flags.
The RXRPC library now terminates calls in the QUEUED state when an ABORT packet is received. This clears the call channel making it available to accept another call and reduces the work load on the worker thread pool.
Fileserver endpoint registration changes no longer result in local invalidation of callbacks from that server.
Receipt of an RXAFSCB_InitCallBackState3 RPC from a fileserver no longer resets the volume site status information for all volumes on all servers.

v2021.05-25 (28 December 2022)

The v2021.05-25 release includes further changes to RXRPC to improve reliability. The changes in this release prevent improper packet size growth. Packet size growth should never occur when a call is attempting to recover from packet loss; and is unsafe when the network path's maximum transmission unit is unknown. Packet size growth with be re-enabled in a future AuriStorFS release that includes Path MTU detection and the Extended SACK functionality.
Improved error text describing the source of invalid values in /etc/yfs/yfs-client.conf or included files and directories.

v2021.05-24 (25 October 2022)

New Platform: macOS 13 (Ventura)

v2021.05-22 (12 September 2022) and v2021.05-21 (6 September 2022)

RX RPC
- If receipt of a DATA packet causes an RX call to enter an error state, do not send the ACK of the DATA packet following the ABORT packet. Only send the ABORT packet.
- AuriStor RX has failed to count and report the number of RX BUSY packets that have been sent. Beginning with this change the sent RX BUSY packet count is once again included in the statistics retrieved via rxdebug server port -rxstats.
- Introduce minimum and maximum bounds checks on the ACK packet trailer fields. If the advertised values are out of bounds for the receiving RX stack, do not abort the call but adjust the values to be consistent with the local RX RPC implementation limits. These changes are necessary to handle broken RX RPC implementations or prevent manipulation by attackers.

New to v2021.05-20 (15 August 2022) and v2021.05-19 (13 August 2022)

RX RPC
- Include the DATA packet serial number in the transmitted reachability check PING ACK. This permits the reachability test ACK to be used for RTT measurement.
- Do not terminate a call due to an idle dead timeout if there is data pending in the receive queue when the timeout period expires. Instead deliver the received data to the application. This change prevents idle dead timeouts on slow lossy network paths.
- Fix assignment of RX DATA, CHALLENGE, and RESPONSE packet serial numbers in macOS (KERNEL). Due to a mistake in the implementation of atomic_add_and_read the wrong serial numbers were assigned to outgoing packets.

New to v2021.05-18 (12 July 2022)

Cache Manager
- Prevent a kernel memory leak of less than 64 bytes for each bulkstat RPC issued to a fileserver. Bulkstat RPCs can be frequently issued and over time this small leak can consume a large amount of kernel memory. Leak introduced in AuriStorFS v0.196.
- The Perl::AFS module directly executes pioctls via the OpenAFS compatibility pioctl interface instead of the AuriStorFS pioctl interface. When Perl::AFS is used to store an access control list (ACL), the deprecated RXAFS_StoreACL RPC would be used in place of the newer RXAFS_StoreACL2 or RXYFS_StoreOpaqueACL2 RPCs. This release alters the behavior of the cache manager to use the newer RPCs if available on the fileserver and fallback to the deprecated RPC. The use of the deprecated RPC was restricted to use of the OpenAFS pioctl interface.
RX RPC
- Handle a race during RX connection pool probes that could have resulted in the wrong RX Service ID being returned for a contacted service. Failure to identify that correct service id can result in a degradation of service.
- The Path MTU detection logic sends padded PING ACK packets and requests a PING_RESPONSE ACK be sent if received. This permits the sender of the PING to probe the maximum transmission unit of the path. Under some circumstances attempts were made to send negative padding which resulted in a failure when sending the PING ACK. As a result, the Path MTU could not be measured. This release prevents the use of negative padding.
Preparation for supporting macOS 13 Ventura when it is released in Fall 2022.
Some shells append a slash to an expanded directory name in response to tab completion. These trailing slashes interfered with "fs lsmount", "fs flushmount" and "fs removeacl" processing. This release includes a change to prevent these commands from breaking when presented a trailing slash.

New to v2021.05-17 (16 May 2022)

Cell Service Database Updates
- Update cern.ch, ics.muni.cz, ifh.de, cs.cmu.edu, qatar.cmu.edu, it.kth.se
- Remove uni-hohenheim.de, rz-uni-jena.de, mathematik.uni-stuttgart.de, stud.mathematik.uni-stuttgart.de, wam.umd.edu
- Add ee.cooper.edu
- Restore ams.cern.ch, md.kth.se, italia
Fix parsing of [afsd] rxwindow configuration which can be used to specified a non-default send/receive RX window size. The current default is 128 packets.
RX Updates
- Add nPacketsReflected

Logging improvements

Cache directory validation errors log messages now include the cache directory path.

Log the active configuration path if "debug" logging is enabled.

More details of rxgk token extraction failures.

New to v2021.05-16 (24 March 2022)

RX - Previous releases re-armed the Retransmission Timeout (RTO) each time a new unacknowledged packet is acknowledged instead of when a new leading edge packet is acknowledged. If leading edge data packet and its retransmission are lost, the call can remain in the "recovery" state where it continues to send new data packets until one of the following is true:
. the maximum window size is reached
. the number of lost and resent packets equals 'cwind'
at which point there is nothing left to transmit. The leading edge data packet can only be retransmitted when entering the "loss" state but since the RTO is reset with each acknowledged packet the call stalls for one RTO period after the last transmitted data packet is acknowledged.

This poor behavior is less noticiable with small window sizes and short lived calls. However, as window sizes and round-trip times increase the impact of a twice lost packet becomes significant.

RX - Never set the high-order bit of the Connection Epoch field. RX peers starting with IBM AFS 3.1b through AuriStor RX v0.191 ignore the source endpoint when matching incoming packets to RX connections if the high-order epoch bit is set. Ignoring the source endpoint is problematic because it can result in a call entering a zombie state whereby all PING ACK packets are immediately responded to the source endpoint of the PING ACK but any delayed ACK or DATA packets are sent to the endpoint bound to the RX connection. An RX client that moves from one network to another or which has a NAT|PAT device between it and the service can find themselves stuck.

Starting with AuriStor RX v0.192 the high-order bit is ignored by AuriStor RX peer when receiving packets. This change to always clear the bit prevents IBM AFS and OpenAFS peers from ignoring the source endpoint.

RX - The initial packetSize calculation for a call is altered to require that all constructed packets before the receipt of the first ACK packet are eligible for use in jumbograms if and only if the local RX stack has jumbograms enabled and the maximum MTU is large enough. By default jumbograms are disabled for all AuriStorFS services. This change will have a beneficial impact if jumbograms are enabled via configuration; or when testing RX performance with "rxperf".

New fs whereis -noresolve option displays the fileservers by network endpoint instead of DNS PTR record hostname.

New to v2021.05-15 (24 January 2022)

kernel - fixed YFS_RXGK service rx connection pool leak

New to v2021.05-14 (20 January 2022)

fs mkmount permit mount point target strings longer than 63 characters.
afsd enhance logging of yfs-rxgk token renewal errors.

afsd gains a "principal = configuration option for use with keytab acquisition of yfs-rxgk tokens for the cache manager identity.

kernel - Avoid unnecessary rx connection replacement by racing threads after token replacement or expiration.

kernel - Fix a regression introduced in v2021.05 where an anonymous combined identity yfs-rxgk token would be replaced after three minutes resulting in the connection switching from yfs-rxgk to rxnull.

kernel - Fix a regression introduced in v0.208 which prevented the invalidation of cached access rights in response to a fileserver callback rpc. The cache would be updated after the first FetchStatus rpc after invalidation.

kernel - Reset combined identity yfs-rxgk tokens when the system token is replaced.

kernel - The replacement of rx connection bundles in the cache manager to permit more than four simultaneous rx calls per uid/pag with trunked rx connections introduced the following regressions in v2021.05.

a memory leak of discarded rx connection objects

failure of NAT ping probes after replacement of an connection

inappropriate use of rx connections after a service upgrade failure

All of these regressions are fixed in patch 14.

New to v2021.05-12 (7 October 2021)

fs ignorelist -type afsmountdir in prior releases could prevent access to /afs.

Location server rpc timeout restored to two minutes instead of twenty minutes.

Location server reachability probe timeout restored to six seconds instead of fifty seconds.

Cell location server upcall results are now cached for fifteen seconds.

Multiple kernel threads waiting for updated cell location server reachability probes now share the results of a single probe.

RX RPC implementation lock hierarchy modified to prevent a lock inversion.

RX RPC client connection reference count leak fixed.

RX RPC deadlock during failed connection service upgrade attempt fixed..

New to v2021.05-9 (25 October 2021)

First public release for macOS 12 Monterey build using XCode 13. When upgrading macOS to Monterey from earlier macOS releases, please upgrade AuriStorFS to v2021.05-9 on the starting macOS release, upgrade to Monterey and then install the Monterey specific v2021.05-9 release.

Improved logging of "afsd" shutdown when "debug" mode is enabled.

Minor RX network stack improvements

New to v2021.05-3 (10 June 2021)

Fix for [cells] cellname = {...} without server list.

New to v2021.05 (31 May 2021)

Multi-homed location servers are finally managed as a single server instead of treating each endpoint as a separate server. The new functionality is a part of the wholesale replacement of the former cell management infrastructure. Location server communication is now entirely managed as a cluster of multi-homed servers for each cell. The new infrastructure does not rely upon the global lock for thread safety.

This release introduces a new infrastructure for managing user/pag entities and tracking their per cell tokens and related connection pools.

Expired tokens are no longer immediately deleted so that its possible for them to be listed by "tokens" for up to two hours.

Prevent a lock inversion introduced in v0.208 that can result in a deadlock involving the GLOCK and the rx call.lock. The deadlock can occur if a cell's list of location servers expires and during the rebuild an rx abort is issued.

Add support for rxkad "auth" mode rx connections in addition to "clear" and "crypt". "auth" mode provides integrity protection without privacy.

Add support for yfs-rxgk "clear" and "auth" rx connection modes.

Do not leak a directory buffer page reference when populating a directory page fails.

Re-initialize state when populating a disk cache entry using the fast path fails and a retry is performed using the slow path. If the data version changes between the attempts it is possible for truncated disk cache data to be treated as valid.

Log warnings if a directory lookup operation fails with an EIO error. An EIO error indicates that an invalid directory header, page header, or directory entry was found.

Do not overwrite RX errors with local errors during Direct-I/O and StoreMini operations. Doing so can result in loss of VBUSY, VOFFLINE, UAENOSPC, and similar errors.

Correct a direct i/o code path which could overwrite a fileserver returned error code with a local RXGEN_CC_UNMARSHAL error.

Correct the StoreMini code path which could overwrite a fileserver returned error code with a local RXGEN_CC_UNMARSHAL error.

Ensure the rx call object is not locked when writing to the network socket.

Removed all knowledge of the KERNEL global lock from RX. Acquiring the GLOCK from RX is never safe if any other lock is held. Doing so is a lock order violation that can result in deadlocks.

Fixed a race in the opr_reservation system that could produce a cache entry reference undercount.

If a directory hash chain contains a circular link, a buffer page reference could be leaked for each traversal.

Each AFS3 directory header and page header contains a magic tag value that can be used in a consistency check but was not previously checked before use of each header. If the header memory is zero filled during a lookup, the search would fail producing an ENOENT error. Starting with this release the magic tag values are validated on each use. An EIO error is returned if there is a tag mismatch.

"fs setcrypt -crypt auth" is now a permitted value. The "auth" mode provides integrity protection but no privacy protection.

Add new "aklog -levels " option which permits requesting "clear" and "auth" modes for use with yfs-rxgk.

Update MKShim to Apple OpenSource MITKerberosShim-79.

Report KLL errors via a notification instead of throwing an exception which (if not caught) will result in process termination.

If an exception occurs while executing "unlog" catch it and ignore it. Otherwise, the process will terminate.

New to v2021.04 (22 April 2021)

Primarily bug fixes for issues that have been present for years.

A possibility of an infinite kernel loop if a rare file write / truncate pattern occurs.

A bug in silly rename handling that can prevent cache manager initiated garbage collection of vnodes.

New to v0.209 (13 March 2021)

fs setserverprefs and fs getserverprefs updated to support IPv6 and CIDR specifications.

Improved error handling during fetch data and store data operations.

Prevents a race between two vfs operations on the same directory which can result in caching of out of date directory contents.

Use cached mount point target information instead of evaluating the mount point's target upon each access.

Avoid rare data cache thrashing condition.

Prevent infinite loop if a disk cache error occurs after the first page in a chunk is written.

Network errors are supposed to be returned to userspace as ETIMEDOUT. Previously some were returned as EIO.

When authentication tokens expire, reissue the fileserver request anonymously. If the anonymous user does not have permission either EACCES or EPERM will be returned as the error to userspace. Previously the vfs request would fail with an RXKADEXPIRED or RXGKEXPIRED error.

If growth of an existing connection vector fails, wait on a call slot in a previously created connection instead of failing the vfs request.

Volume and fileserver location query infrastructure has been replaced with a new modern implementation.

Replace the cache manager's token management infrastructure with a new modern implementation.

New to v0.206 (12 January 2021) - Bug fixes

Prevents a possible panic during unmount of /afs.

Improved failover and retry logic for offline volumes.

New to v0.205 (24 December 2020) - Bug fixes

Volume name-to-id cache improvements

Fix expiration of name-to-id cache entries

Control volume name-to-id via sysctl

Query volume name-to-id statistics via sysctl

Improve error handling for offline volumes

Fix installer to prevent unnecessary installation of Rosetta 2 on Apple Silicon

New to v0.204 (25 November 2020) - Bug fix for macOS Big Sur

v0.204 prevents a kernel panic on Big Sur when AuriStorFS is stopped and restarted without an operating system reboot.

introduces a volume name-to-id cache independent of the volume location cache.

New to v0.203 (13 November 2020) - Bug fix for macOS

v0.203 prevents a potential kernel panic due to network error.

New to v0.201 (12 November 2020) - Universal Big Sur (11.0) release for Apple Silicon and Intel

v0.201 introduces a new cache manager architecture on all macOS versions except for High Sierra (10.12). The new architecture includes a redesign of:

kernel extension load

kernel extension unload (not available on Big Sur)

/afs mount

/afs unmount

userspace networking

The conversion to userspace networking will have two user visible impacts for end users:
The Apple Firewall as configured by System Preferences -> Security & Privacy -> Firewall is now enforced. The "Automatically allow downloaded signed software to receive incoming connections" includes AuriStorFS.

Observed network throughput is likely to vary compared to previous releases.

On Catalina the "Legacy Kernel Extension" warnings that were displayed after boot with previous releases of AuriStorFS are no longer presented with v0.201.

AuriStorFS /afs access is expected to continue to function when upgrading from Mojave or Catalina to Big Sur. However, as AuriStorFS is built specifically for each macOS release, it is recommended that end users install a Big Sur specific AuriStorFS package.
AuriStorFS on Apple Silicon supports hardware accelerated aes256-cts-hmac-sha1-96 and aes128-cts-hmac-sha1-96 using AuriStor's proprietary implementation.

New to v0.200 (4 November 2020) - Final release for macOS El Capitan (10.11)

The network path between a client and a server often traverses one or more network segments separated by NAT/PAT devices. If a NAT/PAT times out a RPCs endpoint translation mid-call, this can result in an extended delay before failure and the server being marked down, or worse, a call that never terminates and a client that appears to hang until the fileserver is restarted.

This release includes significant changes to the RX stack and the UNIX cache manager to detect such conditions, fail the calls quickly and detect when it is safe to retry the RPC.

NAT/PAT devices that drop endpoint mappings while in use are anti-social and can result in unwanted delays and even data loss. Whenever possible they should be avoided when they can be. That said, the changes in this release are a huge step to making the loss of endpoint mappings tolerable.

Fix segmentation fault of Backgrounder when krb5_get_credentials() fails due to lack of network connectivity.

Fix the "afsd" rxbind option which was ignored if the default port, 7001, is in use by another process on the system.

If a direct i/o StoreData or FetchData RPC failed such that it must be retried, the retried RPC would fail due to an attempt to Fetch or Store the wrong amount of data. This is fixed.

Servers are no longer marked down if RPCs fail with RX_CALL_PEER_RESET, RX_CALL_EXCEEDS_WINDOW, or RX_PROTOCOL_ERROR. RPCs that are safe to retry are retried.

Fixed a race between a call entering error state and call completion that can result in a the call remaining in the DALLY state and the connection channel remaining in use. If this occurs during process or system shutdown it can result in a deadlock.

During shutdown cancel any pending delayed aborts to prevent a potential deadlock. If a deadlock occurs when unloading a kernel module a reboot will be required.

Updated cellservdb.conf

New to v0.197.1 (31 August 2020) and v0.198 (10 October 2020)

Prevent Dead vnode has core/unlinkedel/flock panic introduced in v0.197.

New to v0.197 (26 August 2020)

A new callback management framework for UNIX cache managers reduces the expense of processing volume callback RPCs from O(number of vcache objects) to O(1). A significant amount of lock contention has been avoided. The new design reduces the risk of the single callback service worker thread blocking. Delays in processing callbacks on a client can adversely impact fileserver performance and other clients in the cell.

Bulk fetch status RPCs are available on macOS for the first time. Bulk fetch status permits optimistic caching of vnode status information without additional round-trips. Individual fetch status RPCs are no longer issued if a bulk status fails to obtain the required status information.

Hardware accelerated crypto is now available for macOS cache managers. AuriStor's proprietary aes256-cts-hmac-sha1-96 and aes256-cts-hmac-sha512-384 implementations leverage Intel processor extensions: AESNI AVX2 AVX SSE41 SSSE3 to achieve the fastest encrypt, decrypt, sign and verify times for RX packets.

This release optimizes the removal of "._" files that are used to store extended attributes by avoiding unnecessary status fetches when the directory entry is going to be removed.

When removing the final directory entry for an in-use vnode, the directory entry must be silly renamed on the fileserver to prevent removal of the backing vnode. The prior implementation risked blindly renaming over an existing silly rename directory entry.

Behavior change! When the vfs performs a lookup on ".", immediately return the current vnode.

if the object is a mount point, do not perform fakestat and attempt to resolve the target volume root vnode.

do not perform any additional access checks on the vnode. If the caller already knows the vnode the access checks were performed earlier. If the access rights have changed, they will be enforced when the vnode is used just as they would have if the lookup of "." was performed within the vfs.

do not perform a fetch status or fetch data rpcs. Again, the same as if the lookup of "." was performed within the vfs.

Volumes mounted at more than one location in the /afs namespace are problematic on more than one operating system that do not expect directories to have more than one parent. It is particularly problematic if a volume is mounted within itself. Starting with this release any attempt to traverse a mountpoint to the volume containing the mountpoint will fail with ENODEV.

When evaluating volume root vnodes, ensure that the vnode's parent is set to the parent directory of the traversed mountpoint and not the mountpoint. Vnodes without a parent can cause spurious ENOENT errors on Mojave and later.

v0.196 was not publicly released.

New to v0.195 (14 May 2020)

This is a CRITICAL update for AuriStorFS macOS clients.

In Sep 2019 AuriStorFS v0.189 was released which provided faster and less CPU intensive writing of (>64GB) large files to /afs. These improvements introduced a hash collision bug in the store data path of the UNIX cache manager which can result in file corruption. If a hash collision occurs between two or more files that are actively being written to via cached I/O (not direct I/O), dirty data can be discarded from the auristorfs cache before it is written to the fileserver creating a file with a range of zeros (a hole) on the fileserver. This hole might not be visible to the application that wrote the data because the lost data was cached by the operating system. This bug has been fixed in v0.195 and it is for this reason that v0.195 has been designated a CRITICAL release for UNIX/Linux clients.

While debugging a Linux SIGBUS issue, it was observed that receipt of an ICMP network error in response to a transmitted packet could result in termination of an unrelated rx call and could mark a server down. If the terminated call is a StoreData RPC, permanent data loss will occur. All Linux clients derived from the IBM AFS code base experience this bug. The v0.195 release prevents this behavior.

This release includes changes that impact all supported UNIX/Linux cache managers. On macOS there is reduced lock contention between kernel threads when the vcache limit has been reached.

The directory name lookup cache (DNLC) implementation was replaced. The new implementation avoids the use of vcache pointers which did not have associated reference counts, and eliminates the invalidation overhead during callback processing. The DNLC now supports arbitrary directory name lengths; the prior implementation only cached entries with names not exceeding 31 characters.

Prevent matching arbitrary cell name prefixes as aliases. For example "/afs/y" should not be an alias for "your-file-system.com". Some shells, for example "zsh", query the filesystem for names as users type. Delays between typed characters result in filesystem lookups. When this occurs in the /afs dynroot directory, this could result in cellname prefix string matches and the dynamic creation of directory entries for those prefixes.

New to v0.194 (2 April 2020)

This is a CRITICAL release for all macOS users. All prior macOS clients whether AuriStorFS or OpenAFS included a bug that could result in data corruption either when reading or writing.

This release also fixes these other issues:

sign and notarize installer plugin "afscell" bundle. The lack of digital signature prevented the installer from prompting for a cellname on some macOS versions.

prevent potential for corruption when caching locally modified directories.

v0.193 was withdrawn due to a newly introduced bug that could result in data corruption.

New to v0.192 (30 January 2020)

The changes improve stability, efficiency, and scalability. Post-0.189 changes exposed race conditions and reference count errors which can lead to a system panic or deadlock. In addition to addressing these deficiencies this release removes bottlenecks that restricted the number of simultaneous vfs operations that could be processed by the AuriStorFS cache manager. The changes in this release have been successfully tested with greater than 400 simultaneous requests sustained for for several days.

New to v0.191 (16 December 2019)

Restore keyed cache manager capability broken in v0.189.

Add kernel module version string to AuriStorFS Preference Pane.

Other kernel module bug fixes.

New to v0.190 (14 November 2019)

Short-circuit busy volume retries after volume or volume location entry is removed.

New to v0.189 (28 October 2019)

Faster "git status" operation on repositories stored in /afs.

Faster and less CPU intensive writing of (>64GB) large files to /afs. Prior to this release writing files larger than 1TB might not complete. With this release store data throughput is consistent regardless of file size. (See "UNIX Cache Manager large file performance improvements" later in this file).

macOS Catalina (8 October 2019)

AuriStorFS v0.188 released for macOS Catalina (10.15)

New to v0.188 (23 June 2019)

Increased clock resolution for timed waits from 1s to 1ns

Added error handling for rx multi rpcs interrupted by signals

New to v0.186 (29 May 2019)

v0.184 moved the /etc/yfs/cmstate.dat file to /var/yfs. With this change afsd would fail to start if /etc/yfs/cmstate.dat exists but contains invalid state information. This is fixed.

v0.184 introduced a potential deadlock during directory processing. This is fixed.

Handle common error table errors obtained outside an afs_Analyze loop. Map VL errors to ENODEV and RX, RXKAD, RXGK errors to ETIMEDOUT

Log all server down and server up events. Transition events from server probes failed to log messages.

RX RPC networking:

If the RPC initiator successfully completes a call without consuming all of the response data fail the call by sending an RX_PROTOCOL_ERROR ABORT to the acceptor and returning a new error, RX_CALL_PREMATURE_END, to the initiator.
Prior to this change failure to consume all of the response data would be silently ignored by the initiator and the acceptor might resend the unconsumed data until any idle timeout expired. The default idle timeout is 60 seconds.

Avoid transmitting ABORT, CHALLENGE, and RESPONSE packets with an uninitialized sequence number. The sequence number is ignored for these packets but set it to zero.

New to v0.184 (26 March 2019)

The initial congestion window has been reduced from 10 Rx packets to 4. Packet reordering and loss has been observed when sending 10 Rx packets via sendmmsg() in a single burst. The lack of udp packet pacing can also increase the likelihood of transmission stalls due to ack clock variation.

The UNIX Cache Manager underwent major revisions to improve the end user experience by revealing more error codes, improving directory cache efficiency, and overall resiliency. The cache manager implementation was redesigned to be more compatible with operating systems such as Linux and macOS that support restartable system calls. With these changes errors such as "Operation not permitted", "No space left on device", "Quota exceeded", and "Interrupted system call" can be reliably reported to applications. Previously such errors might have been converted to "I/O error".

New to v0.180 (9 November 2018)

RX reliability and performance improvements for high latency and/or lossy network paths such as public wide area networks.

A fix for a macOS firewall triggered kernel panic introduced in v0.177.

New to v0.177 (17 October 2018)

A fix to AuriStor's RX implementation bug introduced in v0.176 that interferes with communication with OpenAFS and IBM Location and File Services.

New to v0.176 (3 October 2018)

AuriStor's RX implementation has undergone a major upgrade of its flow control model. Prior implementations were based on TCP Reno Congestion Control as documented in RFC5681; and SACK behavior that was loosely modelled on RFC2018. The new RX state machine implements SACK based loss recovery as documented in RFC6675, with elements of New Reno from RFC5682 on top of TCP-style congestion control elements as documented in RFC5681. The new RX also implements RFC2861 style congestion window validation.

When sending data the RX peer implementing these changes will be more likely to sustain the maximum available throughput while at the same time improving fairness towards competing network data flows. The improved estimation of available pipe capacity permits an increase in the default maximum window size from 60 packets (84.6 KB) to 128 packets (180.5 KB). The larger window size increases the per call theoretical maximum throughput on a 1ms RTT link from 693 mbit/sec to 1478 mbit/sec and on a 30ms RTT link from 23.1 mbit/sec to 49.39 mbit/sec.

Improve shutdown performance by refusing to give up callbacks to known unreachable file servers and apply a shorter timeout period for the rest.

Permit RXAFSCB_WhoAreYou to be successfully executed after an IBM AFS or OpenAFS fileserver unintentionally requests an RX service upgrade from RXAFSCB to RXYFSCB.

RXAFS timestamps are conveyed in unsigned 32-bit integers with a valid range of 1 Jan 1970 (Unix Epoch) through 07 Feb 2106. UNIX kernel timestamps are stored in 32-bit signed integers with a valid range of 13 Dec 1901 through 19 Jan 2038. This discrepency causes RXAFS timestamps within the 2038-2106 range to display as pre-Epoch.

RX Connection lifecycle management was susceptible to a number of race conditions that could result in assertion failures, the lack of a NAT ping connection to each file server, and the potential reuse of RX connections that should have been discarded.

This release includes a redesigned lifecycle that is thread safe, avoids assertions, prevents NAT ping connection loss, and ensures that discarded connections are not reused.

The 0.174 release unintentionally altered the data structure returned to xstat_cm queries. This release restores the correct wire format.

Since v0.171, if a FetchData RPC fails with a VBUSY error and there is only one reachable fileserver hosting the volume, then the VFS request will immediately with an ETIMEDOUT error ("Connection timed out").

v0.176 corrects three bugs that contributed to this failure condition. One was introduced in v0.171, another in 0.162 and the final one dates to IBM AFS 3.5p1.

The intended behavior is that a cache manager, when all volume sites fail an RPC with a VBUSY error, will sleep for up to 15 seconds and then retry the RPC as if the VBUSY error had never been received. If the RPC continues to receive VBUSY errors from all sites after 100 cycles, the request will be failed with EWOULDBLOCK ("Operation would block") and not ETIMEDOUT.

Prefer VOLMISSING and VOLBUSY error states to network error states when generating error codes to return to the VFS layer. This will result in ENODEV ("No such device") errors when all volume sites return VNOVOL or VOFFLINE errors and EWOULDBLOCK ("Operation would block") errors when all volume sites return VBUSY errors. (v0.176)

New to v0.174 (24 September 2018)

macOS Mojave (10.14) support

New to v0.170 (27 April 2018)

Faster processing of cell configuration information by caching service name to port information.

RX call sequence number rollover to permit calls that require the transmission of more than 5.5TB of data.

Command parser Daylight Saving Time bug fix

Fix a bug that prevented immediate access to a mount point created with "fs mkmount" on the same machine.

Fix the setting of "[afsd] sysnames = " during cache manager startup.

New to v0.168 (6 March 2018)

Corrects "fs setacl -negative" processing [CVE-2018-7168]

Improved reliability for keyed cache managers. More persistent key acquisition renewals.

Major refresh to cellservdb.conf contents.

DNS SRV and DNS AFSDB records now take precedence when use_dns = yes

Kerberos realm hinting provided by
kerberos_realm = [REALM]

DNS host names are resolved instead of reliance on hard coded IP addresses
The cache manager now defaults to sparse dynamic root behavior. Only thiscell and those cells that are assigned aliases are included in /afs directory enumeration at startup. Other cells will be dynamically added upon first access.
Several other quality control improvements.

New to v0.167 (7 December 2017)

Addresses a critical remote denial of service vulnerability [CVE-2017-17432]
Alters the volume location information expiration policy to reduce the risk of single points of failures after volume release operations.
'fs setquota' when issued with quota values larger than 2TB will fail against OpenAFS and IBM AFS file servers
Memory management improvements for the memory caches.

New to v0.164 (11 November 2017)

Internal cache manager redesign. No new functionality.

New to v0.160 (21 September 2017)

Support for OSX High Sierra's new Apple File System (APFS). Customers must upgrade to v0.160 or later before upgrading to OSX High Sierra.
Reduced memory requirements for rx listener thread
Avoid triggering a system panic if an AFS local disk cache file is deleted or becomes inaccessible.
Fixes to "fs" command line output

New to v0.159 (7 August 2017)

Improved failover behavior during volume maintenance operations
Corrected a race that could lead the rx listener thread to enter an infinite loop and cease processing incoming packets.

New to v0.157 (12 July 2017)

Bundled with Heimdal 7.4 to address CVE-2017-11103 (Orpheus' Lyre puts Kerberos to sleep!)
"vos" support for volume quotas larger than 2TB.
"fs flushvolume" works
Fixed a bug that can result in a system panic during server capability testing

New to v0.150

AuriStorFS file server detection improvements

New to v0.149

rxkad encryption is enabled by default. Use "fs setcrypt off" to disable encryption when tokens are available.
Fix a bug in atomic operations on Sierra and El Capitan which could adversely impact Rx behavior.

New to v0.128

Extended attribute ._ files are automatically removed when the associated files are unlinked
Throughput improvements when sending data

New to v0.121

OSX Sierra support

New to v0.117

Cache file moved to a persistent location on local disk
AuriStor File System graphics
Improvements in Background token fetch functionality
Fixed a bug introduced in v0.44 that could result in an operating system crash when enumerating AFS directories containing Unicode file names (v0.106)
El Capitan security changes prevented Finder from deleting files and directories. As of v0.106, the AuriStor OSX client implements the required functionality to permit the DesktopHelperService to securely access the AFS cache as the user permitting Finder to delete files and directories.

Features:

Not vulnerable to OPENAFS-SA-2015-007.
Office 2011 can save to /afs.
Office 2016 can now save files to /afs.
OSX Finder and Preview can open executable documents without triggering a "Corrupted File" warning. .AI, .PDF, .TIFF, .JPG, .DOCX, .XLSX, .PPTX, and other structured documents that might contain scripts were impacted.
All file names are now stored to the file server using Unicode UTF-8 Normalization Form C which is compatible with Microsoft Windows.
All file names are converted to Unicode UTF-8 Normalization Form D for processing by OSX applications.

Known issues:

None

macOS Installer (10.10 Yosemite)

Release Notes

Known Issues

If the Kerberos default realm is not configured, a delay of 6m 59s can occur before the AuriStorFS Backgrounder will acquire tokens and display its icon in the macOS menu. This is the result of macOS performing a Bonjour (MDNS) query in an attempt to discover the local realm.
"aklog" will fail to acquire tokens if the current working directory is located in an inaccessible AFS volume. When the "cwd" is inaccessible, the Heimdal framework is unable to initialize the Kerberos v5 context and reports an out of memory error.

New v2021.05-64 (22 Aug 2025)

The v2021.05-64 release fixes a race condition which can lead to a kernel panic when executing the "unlog" command.

v2021.05-63 (31 May 2025)

When AuriStorFS cache managers added support for per-file ACLs in v0.1, the masking of access rights based upon the inode's S_IRWXU mode bits were skipped when reading objects from AuriStorFS fileservers. The masking continued to be applied when reading objects from OpenAFS fileservers. This release applies S_IRWXU mode bits masking of granted access rights to both AuriStorFS and OpenAFS fileservers.
The existing fake stat mountpoint evaluation logic assumes that the fileserver returned a AFSFetchStatus.ParentVnode that refers to the parent directory from which the mountpoint was located. However, this is only guaranteed to be true if the mountpoint does not have a per-file ACL (which is permitted) and has never been cross-directory hard linked (which is also permitted).
This release improves the error handling logic to fail the fake stat evaluation if the AFSFetchStatus.ParentVnode does not match expectations.
This release fixes a metadata inconsistency when a mountpoint is created when disconnected mode is enabled. A mountpoint should be reported to the vfs as a directory and not a symlink.
The cache manager must track for each inode whether the object's status infomration was received from an AFS3 fileserver or an AuriStorFS fileserver. This release alters when that information is recorded to ensure that it is recorded each time the inode's status information is updated by a fileserver response.
This knowledge is required to properly determine how to interpret the status information.
Directory entries referring to mountpoints and volume roots must not be stored within the Directory Name Lookup Cache (DNLC). Instead, full lookups must be performed each time to ensure proper mountpoint resolution.
Prior to this release locally created mountpoints were stored into the DNLC by mistake.

v2021.05-62 (24 April 2025)

macOS Sequoia v15.4 vs Cisco AnyConnect VPN

Beginning with macOS 15.4, use of Cisco AnyConnect VPN breaks DNS SRV, AFSDB and TXT record lookups performed using the cross-platform DNS resolver API. As a result, clients which rely upon DNS-based zero-configuration resolution of Kerberos realms and/or AFS cells database services are unable to access those resources. This release switches the macOS builds to use the Apple DNS API.
The lookup failures impacted all command line tools, clients and services.
Note: Communication failures can occur if the VPN is configured by the organization to block the transmission of fragmented UDP packets. Set [afsd] rxmaxmtu = 1328 to ensure all RX packets are small enough to traverse the VPN link.
This release improves consistency of vnode metadata interpretation when the status information is received via an RPC which does not return a callback.

v2021.05-61 (31 March 2025)

Instead of allocating the same sized hash table for the vcache and dcache objects regardless of how many elements are expected to be stored, scale the hash table sizes. The vcache hash table will scale from 4096 buckets to 131072 buckets with a target of 4 entries per bucket. The dcache hash tables will scale from 4096 buckets to 65536 buckets with a target of 2 entries per bucket.
Historically, afs cache managers which create symlinks (including mountpoints) locally allocate a fake infinite lifetime callback to permit applications to use the symlink (or mountpoint) without requiring FetchStatus RPCs to be issued to the fileserver. This approach has drawbacks because it means that fileservers will not issue a callback break when the volume containing the symlink is taken offline, moved, deleted, etc. It also means that a callback will not be sent if another cache manager removes the symlink.
Beginning with this release the AuriStorFS cache manager will no longer allocate a fake infinite lifetime callback and will instead rely upon fileserver allocated callbacks.
"fs makemount" is supposed to create a mountpoint even if the target volume does not exist or is inaccessible. v2021.05 introduced support for creating mountpoints which kafs which broke the ability to create mountpoints to non-existent volumes.

v2021.05-60 (6 March 2025)

The cache manager retries fileserver RPCs up to 100 times when the fileserver replies with VBUSY, VOFFLINE or VSALVAGE. The cache manager fails the application request after the limit is reached. Prior to this release, the failure would result in the cache manager returning an ENODEV error to the application. But ENODEV indicates that the required volume does not exist and the fileserver has not indicated that the volume does not exist. The cache manager will now return EWOULDBLOCK to the application because the volume might be temporarily inaccessible.
Many OpenAFS and AuriStorFS fileservers can return VNOVOL as a side effect of a race between a volume going offline so that control can be given to the Volume Management Service (volserver) and an incoming RPC requesting that the volume be attached. A VNOVOL can also be returned because the volume was temporarily deleted from the vice partition prior to being replaced by a new clone, or prior to the fileserver marking the volume group as having been moved to another fileserver.
The fileserver replying to an RPC with VNOVOL cannot be trusted to mean that the volume has been permanently removed unless it is matched by the Location Service reporting the Volume Location (VL) entry has been deleted. From now on when a VNOVOL response is received and the VL entry exists, the cache manager will return EWOULDBLOCK instead of ENODEV to the application.
Prior to this release, any VL error code returned from the Location Service would be interpreted as meaning that the requested VL entry does not exist. Starting with this release only the VL_NOENT error will be interpreted as the VL entry does not exist.
Restrict the internal use of the ENOENT error code so the only use of ENOENT is to indicate that a directory entry does not exist. Previously, ENOENT could be returned if a disk cache entry could not be allocated or found. A disk cache related ENOENT returned when processing a directory might have been confused for the requested directory entry not existing.
Restrict the internal use of the ENODEV error code so that the only meaning of ENODEV is that the required volume does not exist. Previously, ENODEV could be returned due to transient errors when failing to contact the Location Service.

v2021.05-59 (13 February 2025)

The fix for garbage collection of idle unixuser objects introduced in v2021.05-56 was incomplete. As a result, Rx client connections might not be garbage collected.

v2021.05-57 (9 February 2025)

No new changes for macOS.

v2021.05-56 (7 February 2025)

Minor bug fixes to the Rx RPC implementation
The fix for garbage collection of idle unixuser objects introduced in v2021.05-55 was incomplete. As a result, Rx client connections might not be garbage collected.

v2021.05-55 (23 January 2025)

Minor bug fixes to the Rx RPC implementation
During garbage collection of idle unixuser objects, any Rx client connections associated with the unixuser (uid or pag) that were in use by another thread would not have their cache manager reference dropped. As a result the Rx client connections could never be garbage collected which is problematic on systems which churn through many unixusers. The behavior has been changed so that unixusers objects with in use Rx connections will not be freed. This permits future attempts to garbage collect unixusers to do so only after the Rx connections can also be freed.
Although this misbehavior has been present for decades, it is only since v2021.05-52 that the problem has been observed. It is possible that the changes to reduce the holding of the GLOCK has created a window of opportunity for connections to be simultaneously in use during unixuser garbage collection.

v2021.05-54 (19 January 2025)

The UNIX cache manager changes continue AuriStor's efforts to address a report that was received at the end of September. The end user site uses an OpenAFS cell to distribute software for execution by IBM Spectrum LSF (load sharing facility) managed compute nodes. Each of the approximately one dozen compute nodes is configured with approximately 100 processor cores and 1TB of ram. Each job scheduled by LSF runs in a newly allocated Process Authentication Group (PAG). Some jobs run for hours or days; others complete in a few seconds. LSF has been observed scheduling more than 3000 jobs in under a minute with peak counts of cache manager user objects at 1300. Everything ran well until it didn't. After some unexplained trigger event, all of the remaining free memory on the system would be consumed in about an hour causing the system to panic.

Over the last five months AuriStor has diligently worked to develop theories, optimize the AuriStorFS UNIX cache manager to address each theory; then move on to subsequent ideas once prior theories were disproved. Over this time we have replaced the memory management lifecycle for access rights entries and rx events. We have made it possible for Linux permission checks to be processed without acquiring the GLOCK in the majority of access patterns thereby increasing parallelism. The lifecycle management of PAG allocation and user entity tracking has been rewritten to avoid the GLOCK. The Linux inode lifecycle management has been rewritten which some sites attribute to elimination of swapping on their systems. All of these changes increased throughput and parallelism within and without the cache manager. However, none of them prevented the memory consumption once the unknown trigger event occurred.

It is with great pleasure that we can report that the trigger event and the source of the memory consumption have been identified. The trigger is a damaged Rx packet header which causes the AuriStor Rx stack to place an Rx connection into an RX_PROTOCOL_ERROR error state when there is an in-flight call to an OpenAFS fileserver. The memory consumption is due to the infinite loop that occurs when an Rx connection is in an error state and the error code is not one of the error codes which forces replacement of the Rx connection. Within each pass of the loop a structure is allocated to track the history of per-server errors so they can be logged when hard mount mode is enabled for the cell. AuriStor Rx permits up to 12 calls on each Rx connection. As a result there can be up to a dozen processes each spinning and allocating memory when the Rx connection is placed into the error state.

Days before Christmas 2023 AuriStor identified a race condition which can result in damaged Rx DATA packets being written to the network. Initial transmission of Rx DATA packets can be sent either from an application thread or an Rx listener. Packet retransmission occurs either in an Rx listener thread or in the Rx event thread. The race permits a second thread to modify the Rx packet headers while the sendmsg() syscall from the original transmission is executing. This race is possible because the original thread might exhaust its timeslice during the sendmsg() call and it might be many milliseconds before it is rescheduled on a busy system. In the meantime, packet retransmissions might be initiated either by an Rx listener thread processing an ACK packet or the Rx event thread processing a timeout event. If the initial packet transmission completes while the Rx headers are being rewritten the resulting packet sent on the wire might be garbage. AuriStor observed that if the modified packets were jumbograms that silent data corruption could occur so we disabled all use of jumbograms in v2021.05-35 We also notified both IBM and OpenAFS of the problem.

v2021.05-53 modified the RPC result analyze logic to force the replacement of Rx connections in an error state except when the error signals a token expiration. When a token is expired there is no benefit to replacing the connection until new tokens have been acquired. This release further ensures that infinite memory allocation cannot occur by tracking only the most recent error per fileserver; and only does so if hard mount mode is enabled for the cell.

Processes that issued syscalls which became stuck in a RPC retry loop could not be interrupted. In addition to the unintentional retry loops there are intentional retry loops such as those that occur because a fileserver repeated fails the call with a “VBUSY” error. “VBUSY” originally meant that the required volume was temporarily in use by the volume management service (“volserver”) and would be back in service soon so please wait. Beginning with IBM AFS 3.4 the error code was used to force a retry for reasons having nothing to do with the requested volume’s availability. IBM AFS 3.4 introduced the concept of multi-homed clients which meant that the fileserver could no longer track the identity of a cache manager by its IPv4 address. Instead, each cache manager randomly generates a UUID which the fileserver must query each time an incoming RPC is received on a previously unrecognized Rx connection. An Rx connection might be unrecognized because it’s new, or because the client’s IP address changed, or because the fileserver garbage collected its state information. Regardless of the reason, if a call is received and there is no Rx connection binding to a cache manager UUID, then the fileserver application worker thread to which the call was assigned must issue an outgoing RPC to the client’s RXAFSCB service. Each client has at most one worker thread. Prior to this release, every incoming RXAFSCB call required acquisition of the GLOCK before it could execute. If the GLOCK is unavailable, the RXAFSCB thread blocks and no other pending RXAFSCB calls can be processed. If a RXAFSCB call blocks for too long, the fileserver timeouts trigger and the fileserver replies to the incoming RXAFS call with “VBUSY”. “VBUSY” can also be sent if the fileserver’s quota on the number of calls from the same endpoint is exceeded.

The v2021.05-54 release removes the GLOCK requirement for many RXAFSCB RPCs including all of the UUID query RPCs and all whole volume callback RPCs. Callback RPCs for individual vnodes must still acquire the GLOCK. However, it is hoped that these changes will significantly reduce the time required to process the queries generated due to unrecognized Rx connections. The AuriStor Rx stack permits a dozen calls to be issued on a single logical connection which the fileserver sees as three unique Rx connections from the same endpoint.

Each Unix UID or PAG ID must use an independent Rx connection to each fileserver. On a system which processes /afs syscalls for 1200 UID or PAG IDs in a minute, there is the potential for 3600 Rx connections per fileserver. When the fileserver is OpenAFS, this quickly triggers per endpoint throttling and generates VBUSY storms which can result in unlucky client processes experiencing starvation. With this release, starving processes can be interrupted and RXAFSCB UUID queries can be completed without blocking.

AuriStor believes this release will provide significant benefits for large scale compute nodes, web servers and time sharing systems.

v2021.05-53 (4 January 2025)

v2021.05-52 switched the memory allocation strategy for rx_event objects from a global free list to reliance upon the operating system's memory allocator to allocate individual objects which are freed upon rx_event completion or cancelation. Reliance upon the system allocator introduces the risk of allocation failures due to out of memory conditions. If allocation of an rx_event object fails, a reference count leak on the associated rx_connection or rx_call object can occur preventing garbage collection of the object.
This release handles rx_event allocation failures:
- delayed ACK or delayed ABORT events are replaced by immediate transmission of the ACK or ABORT
- other cases fail the call or connection with an RX_OUT_OF_MEMORY error which is translated on the wire to RX_WIRE_CALL_DEAD (-1).
After executing a call to a fileserver, if the Rx connection is in an error state for any reason other than expired tokens, then schedule the Rx connection to be replaced the next time a call is attempted. All calls issued on an Rx connection that is already in an error state fail immediately preventing further communication with the associated fileserver from the UID/PAG.
Rx connections in an expired token error state are not replaced until new tokens are set.
Rx connections can be placed into an error state either due to local or remote failures. Such failures are often generated by the in-use security class (RXKAD/RXGK) but might be due to an out-of-memory condition, system clock advancement or reversal, or even application errors.
This release includes internal code changes to the management of servers, connections and access rights which will be used by a future release to separate PAG ID allocation from the system's UID namespace.
If 'afsd' is configured with a keytab, receipt of a PoweredOn event will force an attempt to acquire system tokens.
On supported macOS versions, enforce the use of code signing identities to authenticate mach messages involving the AFSBackgrounder and the AFSPreference panel.

v2021.05-52 (5 December 2024)

The v2021.05-50 replacement of the linked list implementation of the vcache access cache with a btree and last lookup cache introduced a memory corruption bug which is fixed in this release.
Eliminate the GLOCK requirement for the collection of function call and RPC statistics collection.
Replace small space allocation with a GLOCK free allocator.
Reimplement the cache manager tracing logging routines so they no longer require the GLOCK.
The Rx networking stack schedules many different types of time sensitive events associated with individual packet retransmission, delayed ack and abort processing, path mtu detection, keep alives, and other categories. Prior to this release, all memory allocated to Rx events objects would not be freed to the operating system until the userspace process terminated or the kernel module was unloaded.
Beginning with this release, instead of having a single, global, free list for rxevents, the system allocator is trusted to be able to allocate memory quickly. This simplifies the Rx implementation, and should be more efficient, as most system allocators will have per thread or core slabs, removing the contention of the global lock protecting the rxevent free list. In addition, the change protects against a network event allocating memory that cannot be readily freed.
On non-Total Store Ordering (TSO)architectures, relaxed atomics used as reference counters do not ensure completion of all prior stores or recency of later loads. This release replaces all read-modify-write atomics to be acquire-release, which, among other things, ensures that only the intended ordering is possible.

v2021.05-51 (26 November 2024)

Fix a vcache (aka inode) reference leak that can occur during an access rights query for a non-directory object if the neither the calling process nor the cache manager possesses tokens for the cell containing the object. The leak was introduced in v2021.05-45.
When scanning the token list for each user searching for expired tokens the user spinlock must be held to prevent races. The updated user management implementation pulled into v2021.05-45 failed to hold the spinlock.
During cache manager module shutdown reset global structures to the module initialization state. Doing so improves the likelihood that the module can be reinitialized.

v2021.05-50 (23 November 2024)

The linked list implementation of the vcache access rights cache was replaced with a btree and a last lookup cache. The btree design is more efficient for lookups and removals, and scales better as the number of uids and/or process authentication groups (pags) accessing /afs increases.
Prune access rights for expired users. Doing so reduces the cost of subsequent lookups.
Cache the access rights obtained for a uid|pag via the "anyAccess" field. Previously, a uid|pag that obtained its rights from the "anyAccess" field had to perform and fail a rights lookup on each and every vcache access.
When setting or clearing tokens for a single cell, no longer invalidate the access rights cached for the uid|pag for all cells. Instead, only invalidate the access rights cached for the cell whose tokens were set or cleared.

v2021.05-49 (16 November 2024)

The output of "tokens" command failed to report yfs-rxgk tokens was broken starting in v2021.05-46

v2021.05-48 (12 November 2024)

Preallocated buffer overflows in XDR responses (CVE-2024-10397)
The AuriStorFS and AFS3 RPC suites rely upon Sun RPC XDR to marshal binary data structures for network transfer. The AuriStor XDR implementation is derived from Sun Microsystems' Sun RPC code base. The Sun RPC XDR API permits memory for output parameters to (optionally) be preallocated which can result in various classes of memory corruption and/or memory leaks in RPC initiator processes.
The AuriStorFS v2021.05-48 release introduces additional data length validation checks within the AuriStor XDR implementation and prohibits the use of preallocated memory for string output parameters or fields. All cache managers, servers and command line tools are modified by these changes.

v2021.05-46 (28 October 2024)

Cache Manager:
- Prevent a kernel memory leak when server preferences are set via the yfs-client.conf [afsd] configuration or via "fs setserverprefs".
- Directory enumeration of a truncated directory now returns an error instead of assuming the end of the directory has been reached.
- Since AFS 3.0, the Unix cache manager has used the root identity credentials to create anonymous outgoing connections to the location service and each fileserver. However, if uid 0 is assigned a token, then those Rx connections will no longer be anonymous. Beginning with this release anonymous outgoing connections are always created with the NOPAG identity (uid 0xffffffff) instead of the root identity.
- When establishing an outgoing rxgk connection, do not fallback to the systemuser's credentials if the user's credentials resulted in a fatal error. Falling back to the systemuser's credentials can result in inappropriate use of an anonymous connection.
- Improved access rights cache correctness for YFS servers
  In prior releases, the access check logic used the file rights for any files fetched from an AuriStorFS fileserver. For files fetched from an AFS-3 fileserver (and, historically, for all files), it used the directory rights, with the (a)dmin right from the file mixed in. The (a)dmin right on a non-directory indicates that the object is owned by the authenticated user.
  This approach has some issues when combined with the access rights cache, and current fileserver callback behavior. On an AuriStorFS file server, the rights on a non-directory may be determined by the rights granted on its parent directory or, with per-file ACLs, those granted on the object itself. The fileserver will only break a non-directory's callback when a per-file ACL is changed - changing a directory ACL will not break callbacks on files within that directory. This means that changing a directory ACL will not invalidate access rights cache entries on files in tha directory, even if the effective ACL on this files has changed, and the cached rights are no longer correct.
  This release works around this by adding a new function which returns the access rights for a file hosted on an AuriStor fileserver. It uses the parent vnode information to locate the parent directory. If the parent directory isn't in the cache, or it doesn't have a valid callback, or if it has been changed since the file's access rights were cached, it clears the current access rights. Files without a parent directory must have per-file ACLs, and so their cached rights can be safely used.
  Note that files with parent vnodes may still have per-file ACLs, and that the breadcrumbing performed by the client may add parent vnode fields to vnodes which don't have them provided by the fileserver. Such vnodes may have their cached access rights cleared more frequently than necessary.
- Add a new mechanism for caching access rights within the vcache structure. This cache is protected via a vcache specific spinlock, and can be accessed without holding the GLOCK.
  This new cache mechanism returns the memory associated with cached rights back to the kernel's slab free memory pool instead of adding the unused rights structures to a cache manager managed free list. The previous cache implementation never returned allocated memory to the kernel. Instead, invalidated access rights were appended to a free access rights queue for later reuse.
- When a volume is accessed via multiple mountpoints a choice must be made regarding which mountpoint is considered to be the active (or parent) mountpoint. This release alters the behavior such that the active mountpoint is set every time a mountpoint is traversed.
  This behavior is easier to understand and is more likely to provide the expected result for a single process that repeatedly accesses volumes from multiple mountpoints. However, it can result in unexpected results when multiple processes are traversing multiple mountpoints in parallel without any synchronization.

v2021.05-45 (Not released)

v2021.05-44a (18 September 2024)

Authentication:
- AuriStorFS v2021.05-44 included an updated version of the Heimdal Kerberos framework used by AuriStorFS when acquiring yfs-rxgk and rxkad authentication tokens. The updated Heimdal included a bug which disabled the use of DNS SRV records for KDC discovery and DNS TXT records for realm discovery. As a side effect token acquisition might fail with an unable to reach any KDC in realm error. This is fixed in v2021.05-44a.

v2021.05-44 (17 August 2024)

Cache Manager:
- Since v0.192 the cache manager has failed acquire the global lock when upgrading a shared-lock to a write-lock during the execution of a background cache chunk file truncation.
Authentication:
- Neither MIT nor Heimdal gssapi nor their gss mechanisms consistently initialize the output 'minorStatus' parameter. Various functions can return either success or failure majorStatus values with minorStatus unassigned. As a result, stack garbage will be used when generating error messages. From now on libyfs_acquire will always initialize the minorStatus output variable to zero before calling into the gssapi library.
Command Parser:
- No longer accept the token "-" as a switch which eventually fails with a CMD_UNKNOWNSWITCH error. Instead, process the token as a data value.
- Optimize the processing of the loop which processes "source" command input.
- If the source command input file is "-", read from stdin.

v2021.05-41 (26 June 2024)

Rx Networking (libyfs_rx):
- A race during event creation can lead to the freeing of the event while its still in use.
- RFC1122 says that Net and Host unreachable ICMP errors might be transient and should therefore not be treated as fatal. There is no such language for the equivalent ICMPV6 errors. However, in practice ICMP6_DST_UNREACH_NOROUTE, ICMP6_DST_UNREACH_BEYONDSCOPE, and ICMP6_DST_UNREACH_ADDR can be transient.
  Linux has considered these ICMPV6 destination unreachable errors as non-fatal going back at least as far as the initial git repository commit.
  AuriStor Rx has always treated these as fatal errors which results in immediate termination of in-flight calls when received. Even if the network route corrects itself before the call timeout period expires. This release mirrors the Linux behavior and makes these errors non-fatal.
Cache Manager:
- For the first time the cache manager can detect the deletion of a volume and handle the creation of a new volume with the same name but a different volume id.
- If the location service reports the deletion of a volume, invalidate all mount points to that volume.
- RXAFS_GetCapabilities RPC failures should not be treated as a fatal error preventing failover to another replica site.
Authentication ("libyfs_acquire") used by aklog, vos, pts, bos, afsio:
- rxkad_k5 token acquisition krb5 ccache management. This release altered the krb5 credential cache management strategy once again to work around different bugs in MIT krb5 and Heimdal.
- New ACQUIRE_ERR_CRED_EXPIRED error code introduced to represent the case when a request for a service credential returns one that is already expired.
Command parser (libyfs_cmd):
- When parsing configuration files there is a depth limit of ten active inclusions. This limit was improperly enforced as a limit of ten included files instead of a depth of ten included files. As of this release it is now possible to populate an includedir directory with any number of .conf files.

v2021.05-40

Not released.

v2021.05-39 (20 May 2024)

Parallel Random Number Generation:
- AuriStorFS processes rely upon the krb5_generate_random() and RAND_bytes() functions to obtain random bytes for cryptographic operations and random counters. krb5_generate_random() internally acquires a mutex to protect internal state information. This mutex has become a significant barrier to the encryption and checksumming of Rx packets with both yfs-rxgk and rxkad.
  
  This release replaces general use of krb5_generate_random() and RAND_bytes() with a per-thread ChaCha20 CS-PRNG. This avoids the acquisition of a global mutex and permits increased parallelism on multi-core systems.
Rx Networking (libyfs_rx):
- The Rx network stack schedules a garbage collection operation to execute once per minute. This operation enforces call timeouts, destroys idle connections and destroys idle peers. The operation has historically been performed by the Rx event thread which is already responsible for performing actions in response to call RTOs, sending NAT Ping and keep-alive packets, and retrying connection challenge and reachability checks.
  
  The time complexity of the garbage collection operation is determined by the number of calls, connections, and peers. The busier the Rx endpoint the more work must be performed during each garbage collection run and the longer it takes to complete. While garbage collection is active other events cannot be processed which can interfere with the proper flow control of active calls.
  
  As with all Rx events, the garbage collection event is scheduled to execute at an absolute clock time. If the system clock drifts (or is administratively set) backwards garbage collection will not be performed until the clock catches up with the scheduled time.
  
  Another responsibility of the garbage collection procedure is to terminate calls if the system clock drifted backwards by five minutes or longer. However, when the clocked drifts backwards garbage collection is not performed until the clock has advanced beyond the point where calls require termination. As a result, calls are not terminated due to backwards clock drift and they can stall.
  
  This release re-implements the garbage collection procedure using a dedicated thread and relative waits. This change ensures that the garbage collection procedure will not prevent the execution of call related events and permits calls to be terminated when large backward clock drifts are detected.
Disk Cache Management:
- Since IBM AFS 3.5, the cache has been considered "too full" even if there exist cache files have been discarded but not yet truncated. When the cache is "too full" most operations that write to the cache will block until truncation of discarded cache files has been performed which results in unnecessary delays. This release fixes the cache such that that discarded but not yet truncated cache files do not block write operations.
- This release permits the cache truncation daemon thread to exit sooner if the cache manager is shutting down.
Improved failover when the RXGK service (co-located with each vlserver) fails to issue tokens. The failures might be the result of misconfiguration, an inability to read keys or loss of Ubik quorum.

v2021.05-38 (29 February 2024)

As with other AuriStorFS releases since the beginning of 2024, this release includes additional improvements to the Rx RPC implementation which are related to the possibility of silent data corruption when Rx jumbograms are in use. Prior releases disabled the negotiation of Rx jumbograms such that the v2021.05-37 Rx peer will refuse to send Rx jumbograms and will request that the remote peer does not send them. However, a bad actor could choose to send Rx jumbograms even though they were asked not to. v2021.05-38 introduces additional protections to ensure that a corrupt Rx jumbogram is dropped instead of being accepted.

The v2021.05-38 Rx RPC implementation also includes two optimizations. First, when Rx initiators complete a call they will no longer send an extra ACK packet to the Rx acceptor of the completed call. The sending of this unnecessary ACK creates additional work for the server which can result in increased latency for other calls being processed by the server.

Second, all AuriStor Rx services require a reach check for incoming calls from Rx peers to help protect against Distributed Reflection Denial of Service (DRDoS) attacks and execution of RPCs when the response cannot be delivered to the caller. A new reach check is required for each new call that arrives more than 60 seconds after the prior reach check completed. v2021.05-38 Rx considers the successful acknowledgment of a response DATA packet as a reach check validation. With this change reach checks will not be periodically required for a peer that completes at least one call per 60 seconds. A 1 RTT delay is therefore avoided each time a reach check can be avoided. In addition, reach checks require the service to process an additional ACK packet. Eliminating a large number of reach checks can improve overall service performance.

The final Rx RPC change in this release is specific to kernel implementations. Prior releases restricted the frequency of executing time scheduled Rx events to a granularity no smaller than 500ms. As a result an RTO timer event for a lost packet could not be shorter than 500ms even if the measured RTT for the connection is significantly smaller. The minimum RTO for a connection in AuriStor Rx is 200ms. The inability to schedule shorter timeouts impacts recovery from packet loss.

v2021.05-37 (5 February 2024)

Rx improvements:
- The v2021.05-36 release permanently disabled all use of Rx jumbograms due to a risk of silent data corruption. However, when advertising the number of acceptable datagrams in the ACK trailer a missing htonl() set the value to 16777216 instead of 1 on little-endian systems.
- When sending a PING ACK as a reachability test, ensure that the previousPacket field is properly assigned to the largest accepted DATA packet sequence number instead of zero.
- Replace the initialization state flag with two flags. One that indicates that Rx initialization began and the other that it succeeded. The first prevents multiple attempts at initialization after failure. The second prevents shutdown from accessing uninitialized structures if initialization failed.
Cache Manager Improvements:
- No longer refuse to start if both the 'cachedir' and 'memcache' options are present in the configuration file.
- New variable to store the maximum number of cache blocks used. which is accessible via /proc/fs/auristorfs/cache/blocks_used_max.

v2021.05-36 (10 January 2024)

Rx improvements:
- Permanently disable all use of Rx jumbograms due to a risk of silent data corruption.
  
  Ever since OpenAFS 1.0, and possibly before, a race condition has existed when Rx transmits packets. As the rx_call.lock is dropped when starting packet transmission, there is no protection for data that is being copied into the kernel by sendmsg(). It is critical that this packet data is not modified by another thread. However, races exist between the application, listener, and event threads that can lead to retransmissions starting whilst an original transmission is still in progress. This can lead to the packet headers being overwritten, and either the original transmission, the retransmission or both sending corrupt data to the peer.
  
  This corruption can affect the packet serial number or packet flags. It is particularly harmful when the packet flags are corrupted, as this can lead to multiple Rx packets which were intended to be sent as Rx jumbograms being delivered and misinterpreted as a single large packet. The eventual result of this depends on the Rx security class in play, but it can cause decrypt integrity errors (rxgk:crypt and rxgk:auth) or corruption of the data stream (rxnull, rxgk:clear or rxkad:auth).
  
  All AuriStorFS servers, OpenAFS 1.6 or later servers, and the Windows cache manager have been shipped with Rx jumbograms disabled by default. The UNIX cache managers however are shipped with jumbograms enabled. There are many AFS cells around the world that continue to deploy OpenAFS 1.4 or earlier fileservers which continue to negotiate the use of Rx jumbograms.
  
  It is worth noting that all AuriStorFS v0.198 and later fileservers and cache managers implement explicit checks that will recognize the corrupted application data stream and prevent corrupted file content from being stored either into an AFS vnode's backing store in the volume's object store or the cache manager's AFS cache. OpenAFS cache managers and fileservers do not have these protections.
  
  With Rx jumbograms disabled the maximum number of Rx packets in a datagram is reduced from 6 to 1; the maximum number of send and receive datagram fragments is reduced from 4 to 1; and the maximum advertised MTU is restricted to 1444 - the maximum rx packet size prior to the introduction of jumbograms in IBM AFS 3.5.
- If the rx call flow state transitions from either the RECOVERY or RESCUE states to the LOSS state as a result of an RTO resend event while writing packets to the network, cease transmission of any new DATA packets if there are packets in the resend queue.
- When the call flow state is LOSS and all packets in the resend queue have been retransmitted and yet the recovery point has not been reached, then permit new DATA packets to be sent in order to maintain a full congestion window.
- Add a safety check to prevent the estimated RTT from underflowing when the actual roundtrip time is smaller than 125us.
- Fix the computation of the padding required for rxgk encrypted packets. This bug results in packets sending 8 bytes fewer per packets than the network permits. This bug accidentally prevented the construction of Rx jumbograms when a call is protected by rxgk:crypt.
- Replace the random number generator with a more security source of random bytes.

v2021.05-33 (27 November 2023)

Rx improvements:
- Not all calls transfer enough data to be able to measure a smoothed round-trip time (SRTT). Calls which are unable to compute a SRTT should not be used to update the peer host RTO value which is used to initialize the RTO for subsequent calls.
  
  Without this change, a single DATA packet call will cause the peer host RTO to be reduced to 0ms. Subsequent calls will start with a RTO value of MAX(0, rxi_minPeerTimeout) where rxi_minPeerTimeout defaults to 200ms. If the actual measured RTO is greater than 200ms, then initial RTO will be too small resulting in premature triggering of the RTO timer and the call flow state entering the loss phase which can significantly hurt performance.
- Initialize the peer host RTO to rxi_minPeerTimeout (which defaults to 200ms) instead of one second. Although RFC6298 recommends the use of one second when no SRTT is available, Rx has long used the rxi_minPeerTimeout value for other purposes which are supposed to be consistent with initial RTO value. It should be noted that Linux TCP uses 200ms instead of one second for this purpose.
- If associating a security class with an Rx connection fails immediately place the Rx connection into an error state. A failure might occur if the security class is unable to access valid key material.
- If an incoming Rx call requires authentication and the security class is unable to successfully generate a challenge, put the incoming Rx connection into an error state and issue an abort to the caller.
- If an incoming Rx call requires authentication and the security class is able to generate a challenge but the challenge cannot be returned to Rx, then treat this as a transient error. Do not acknowledge the incoming DATA packet and do not place the Rx connection into an error state. An attempt to re-issue the challenge will be performed when the DATA packet is retransmitted.
- If an Rx call is terminated due to the expiration of the configured connection dead time, idle dead time, hard dead time, or as a result of clock drift, then send an ABORT to the peer notifying them that the call has been terminated. This is particularly important for terminated outgoing calls. If the peer does not know to terminate the call, then the call channel might be in use when the next outgoing call is issued using the same call channel. If the next incoming call is received by an in-use call channel, the receiver must drop the received DATA packet and return a BUSY packet. The call initiator will need to wait for a retransmission timeout to pass before retransmitting the DATA packet. Receipt of BUSY packets cannot be used to keep a call alive and therefore the requested call is at greater risk of timing out if the network path is congested.
aklog and krb5.log (via libyfs_acquire):
- If the linked Kerberos library implements krb5_cc_cache_match() and libacquire has been told to use an explicit principal name and credential cache, the Kerberos library might return KRB5_CC_NOTFOUND even though the requested credential cache is the correct one to use. This release will not call krb5_cc_cache_match() if the requested credential cache contains the requested principal.
Cell Service Database (cellservdb.conf):
- cellservdb.conf has been synchronized with the 31 Oct 2023 update to the grand.central.org CellServDB file.

v2021.05-32 (9 October 2023)

No significant changes for macOS compared to v2021.05-31

v2021.05-31 (25 September 2023)

New platform:
- macOS 14 Sonoma
macOS 14 Sonoma:

AuriStorFS v2021.05-29 and later installers for macOS 13 Ventura are compatible with macOS 14 Sonoma and do not need to be removed before upgrading to macOS 14 Sonoma. Installation of the macOS 14 Sonoma version of AuriStorFS is recommended.

Cache Manager:
- If an AuriStorFS cache manager is unable to use the yfs-rxgk security class when communicating with an AuriStorFS fileserver, it must assume it is IBM AFS 3.6 or OpenAFS and upgrade it to AuriStorFS if an upgrade probe returns a positive result. Once a fileserver's type is identified as AuriStorFS the type should never be reset; even if communication with the fileserver is lost or the fileserver restarts.
  
  If an AuriStorFS fileserver is replaced by an OpenAFS fileserver on the same endpoint, then the UUID of the OpenAFS must be different. As a result, the OpenAFS fileserver will be observed as distinct from the AuriStorFS fileserver that previously shared the endpoint.
  
  Prior to this release there were circumstances in which the cache manager discarded the fileserver type information and would fail to recognize the fileserver as an AuriStorFS fileserver when yfs-rxgk could not be used. This release prevents the cache manager from resetting the type information if the fileserver is marked down.
- If a fileserver's location service entry is updated with a new uniquifier value (aka version number), this indicates that one of the following might have changed:
  1. the fileserver's capabilities
  2. the fileserver's security policy
  3. the fileserver's knowledge of the cell-wide yfs-rxgk key
  4. the fileserver's endpoints
  Beginning with this release the cache manager will force the establishment of new Rx connections to the fileserver when the uniquifier changes. This ensures that the cache manager will attempt to fetch new per-fileserver yfs-rxgk tokens from the cell's RXGK service, enforce the latest security policy, and not end up in a situation where its existing tokens cannot be used to communicate with the fileserver.
aklog:
- Fix incorrect output when populating the server list for a service fails. The stashed extended error explaining the cause of the failure was not displayed.
- If a cell has neither _afs3-prserver._udp. DNS SRV records nor AFSDB records, the lookup of the cell's protection servers would fail if there is no local cell configuration details. The fallback to use _afs3-vlserver._udp. DNS SRV records did not work. This is corrected in this release.

v2021.05-30 (6 September 2023)

Do not mark a fileserver down in response to a KRB5 error code.
fs cleanacl must not store back to the file server a cleaned acl if it was inherited from a directory. Doing so will create a file acl.
Correct the generation of never expire rxkad_krb5 tokens from Kerberos v5 tickets which must have a start time of Unix epoch and an end time of 0xFFFFFFFF seconds. The incorrectly generated tokens were subject to the maximum lifetime of 30 days.
Correct the generation of the yfs-rxgk RESPONSE packet header which failed to specify the key version generation number used to encrypt the authenticator. If the actual key version is greater than zero, then the authenticator would fail to verify.
Enforce a maximum NAT ping period of 20s to ensure that NAT/PAT/firewall rules due not expire while Rx RPCs are in-flight.

v2021.05-29 (26 June 2023)

Execution of fs commands such as examine, whereis, listquota, fetchacl, cleanacl, storeacl, whoami, lsmount, bypassthreshold and getserverprefs could result in memory leaks by the AuriStorFS kernel extension.

v2021.05-27 (1 May 2023)

Fixes for bugs in vos introduced in v2021.05-26.

v2021.05-26 (17 April 2023)

Fixed a potential kernel memory leak when triggered by fs examine, fs listquota, or fs quota.
Increased logging of VBUSY, VOFFLINE, VSALVAGE, and RX_RESTARTING error responses. A log message is now generated whenever a task begins to wait as a result of one of these error responses from a fileserver. Previously, a message was only logged if the volume location information was expired or discarded.
Several changes to optimize internal volume lookups.
Faster failover to replica sites when a fileserver returns RX_RESTARTING, VNOVOL or VMOVED.
rxdebug regains the ability to report rx call flags and rx_connection flags.
The RXRPC library now terminates calls in the QUEUED state when an ABORT packet is received. This clears the call channel making it available to accept another call and reduces the work load on the worker thread pool.
Fileserver endpoint registration changes no longer result in local invalidation of callbacks from that server.
Receipt of an RXAFSCB_InitCallBackState3 RPC from a fileserver no longer resets the volume site status information for all volumes on all servers.

v2021.05-25 (28 December 2022)

The v2021.05-25 release includes further changes to RXRPC to improve reliability. The changes in this release prevent improper packet size growth. Packet size growth should never occur when a call is attempting to recover from packet loss; and is unsafe when the network path's maximum transmission unit is unknown. Packet size growth with be re-enabled in a future AuriStorFS release that includes Path MTU detection and the Extended SACK functionality.
Improved error text describing the source of invalid values in /etc/yfs/yfs-client.conf or included files and directories.

v2021.05-24 (25 October 2022)

New Platform: macOS 13 (Ventura)

v2021.05-22 (12 September 2022) and v2021.05-21 (6 September 2022)

RX RPC
- If receipt of a DATA packet causes an RX call to enter an error state, do not send the ACK of the DATA packet following the ABORT packet. Only send the ABORT packet.
- AuriStor RX has failed to count and report the number of RX BUSY packets that have been sent. Beginning with this change the sent RX BUSY packet count is once again included in the statistics retrieved via rxdebug server port -rxstats.
- Introduce minimum and maximum bounds checks on the ACK packet trailer fields. If the advertised values are out of bounds for the receiving RX stack, do not abort the call but adjust the values to be consistent with the local RX RPC implementation limits. These changes are necessary to handle broken RX RPC implementations or prevent manipulation by attackers.

New to v2021.05-20 (15 August 2022) and v2021.05-19 (13 August 2022)

RX RPC
- Include the DATA packet serial number in the transmitted reachability check PING ACK. This permits the reachability test ACK to be used for RTT measurement.
- Do not terminate a call due to an idle dead timeout if there is data pending in the receive queue when the timeout period expires. Instead deliver the received data to the application. This change prevents idle dead timeouts on slow lossy network paths.
- Fix assignment of RX DATA, CHALLENGE, and RESPONSE packet serial numbers in macOS (KERNEL). Due to a mistake in the implementation of atomic_add_and_read the wrong serial numbers were assigned to outgoing packets.

New to v2021.05-18 (12 July 2022)

Cache Manager
- Prevent a kernel memory leak of less than 64 bytes for each bulkstat RPC issued to a fileserver. Bulkstat RPCs can be frequently issued and over time this small leak can consume a large amount of kernel memory. Leak introduced in AuriStorFS v0.196.
- The Perl::AFS module directly executes pioctls via the OpenAFS compatibility pioctl interface instead of the AuriStorFS pioctl interface. When Perl::AFS is used to store an access control list (ACL), the deprecated RXAFS_StoreACL RPC would be used in place of the newer RXAFS_StoreACL2 or RXYFS_StoreOpaqueACL2 RPCs. This release alters the behavior of the cache manager to use the newer RPCs if available on the fileserver and fallback to the deprecated RPC. The use of the deprecated RPC was restricted to use of the OpenAFS pioctl interface.
RX RPC
- Handle a race during RX connection pool probes that could have resulted in the wrong RX Service ID being returned for a contacted service. Failure to identify that correct service id can result in a degradation of service.
- The Path MTU detection logic sends padded PING ACK packets and requests a PING_RESPONSE ACK be sent if received. This permits the sender of the PING to probe the maximum transmission unit of the path. Under some circumstances attempts were made to send negative padding which resulted in a failure when sending the PING ACK. As a result, the Path MTU could not be measured. This release prevents the use of negative padding.
Preparation for supporting macOS 13 Ventura when it is released in Fall 2022.
Some shells append a slash to an expanded directory name in response to tab completion. These trailing slashes interfered with "fs lsmount", "fs flushmount" and "fs removeacl" processing. This release includes a change to prevent these commands from breaking when presented a trailing slash.

New to v2021.05-17 (16 May 2022)

Cell Service Database Updates
- Update cern.ch, ics.muni.cz, ifh.de, cs.cmu.edu, qatar.cmu.edu, it.kth.se
- Remove uni-hohenheim.de, rz-uni-jena.de, mathematik.uni-stuttgart.de, stud.mathematik.uni-stuttgart.de, wam.umd.edu
- Add ee.cooper.edu
- Restore ams.cern.ch, md.kth.se, italia
Fix parsing of [afsd] rxwindow configuration which can be used to specified a non-default send/receive RX window size. The current default is 128 packets.
RX Updates
- Add nPacketsReflected

Logging improvements

Cache directory validation errors log messages now include the cache directory path.

Log the active configuration path if "debug" logging is enabled.

More details of rxgk token extraction failures.

New to v2021.05-16 (24 March 2022)

RX - Previous releases re-armed the Retransmission Timeout (RTO) each time a new unacknowledged packet is acknowledged instead of when a new leading edge packet is acknowledged. If leading edge data packet and its retransmission are lost, the call can remain in the "recovery" state where it continues to send new data packets until one of the following is true:
. the maximum window size is reached
. the number of lost and resent packets equals 'cwind'
at which point there is nothing left to transmit. The leading edge data packet can only be retransmitted when entering the "loss" state but since the RTO is reset with each acknowledged packet the call stalls for one RTO period after the last transmitted data packet is acknowledged.

This poor behavior is less noticiable with small window sizes and short lived calls. However, as window sizes and round-trip times increase the impact of a twice lost packet becomes significant.

RX - Never set the high-order bit of the Connection Epoch field. RX peers starting with IBM AFS 3.1b through AuriStor RX v0.191 ignore the source endpoint when matching incoming packets to RX connections if the high-order epoch bit is set. Ignoring the source endpoint is problematic because it can result in a call entering a zombie state whereby all PING ACK packets are immediately responded to the source endpoint of the PING ACK but any delayed ACK or DATA packets are sent to the endpoint bound to the RX connection. An RX client that moves from one network to another or which has a NAT|PAT device between it and the service can find themselves stuck.

Starting with AuriStor RX v0.192 the high-order bit is ignored by AuriStor RX peer when receiving packets. This change to always clear the bit prevents IBM AFS and OpenAFS peers from ignoring the source endpoint.

RX - The initial packetSize calculation for a call is altered to require that all constructed packets before the receipt of the first ACK packet are eligible for use in jumbograms if and only if the local RX stack has jumbograms enabled and the maximum MTU is large enough. By default jumbograms are disabled for all AuriStorFS services. This change will have a beneficial impact if jumbograms are enabled via configuration; or when testing RX performance with "rxperf".

New fs whereis -noresolve option displays the fileservers by network endpoint instead of DNS PTR record hostname.

New to v2021.05-15 (24 January 2022)

kernel - fixed YFS_RXGK service rx connection pool leak

New to v2021.05-14 (20 January 2022)

fs mkmount permit mount point target strings longer than 63 characters.
afsd enhance logging of yfs-rxgk token renewal errors.

afsd gains a "principal = configuration option for use with keytab acquisition of yfs-rxgk tokens for the cache manager identity.

kernel - Avoid unnecessary rx connection replacement by racing threads after token replacement or expiration.

kernel - Fix a regression introduced in v2021.05 where an anonymous combined identity yfs-rxgk token would be replaced after three minutes resulting in the connection switching from yfs-rxgk to rxnull.

kernel - Fix a regression introduced in v0.208 which prevented the invalidation of cached access rights in response to a fileserver callback rpc. The cache would be updated after the first FetchStatus rpc after invalidation.

kernel - Reset combined identity yfs-rxgk tokens when the system token is replaced.

kernel - The replacement of rx connection bundles in the cache manager to permit more than four simultaneous rx calls per uid/pag with trunked rx connections introduced the following regressions in v2021.05.

a memory leak of discarded rx connection objects

failure of NAT ping probes after replacement of an connection

inappropriate use of rx connections after a service upgrade failure

All of these regressions are fixed in patch 14.

New to v2021.05-12 (7 October 2021)

fs ignorelist -type afsmountdir in prior releases could prevent access to /afs.

Location server rpc timeout restored to two minutes instead of twenty minutes.

Location server reachability probe timeout restored to six seconds instead of fifty seconds.

Cell location server upcall results are now cached for fifteen seconds.

Multiple kernel threads waiting for updated cell location server reachability probes now share the results of a single probe.

RX RPC implementation lock hierarchy modified to prevent a lock inversion.

RX RPC client connection reference count leak fixed.

RX RPC deadlock during failed connection service upgrade attempt fixed..

New to v2021.05-9 (25 October 2021)

First public release for macOS 12 Monterey build using XCode 13. When upgrading macOS to Monterey from earlier macOS releases, please upgrade AuriStorFS to v2021.05-9 on the starting macOS release, upgrade to Monterey and then install the Monterey specific v2021.05-9 release.

Improved logging of "afsd" shutdown when "debug" mode is enabled.

Minor RX network stack improvements

New to v2021.05-3 (10 June 2021)

Fix for [cells] cellname = {...} without server list.

New to v2021.05 (31 May 2021)

Multi-homed location servers are finally managed as a single server instead of treating each endpoint as a separate server. The new functionality is a part of the wholesale replacement of the former cell management infrastructure. Location server communication is now entirely managed as a cluster of multi-homed servers for each cell. The new infrastructure does not rely upon the global lock for thread safety.

This release introduces a new infrastructure for managing user/pag entities and tracking their per cell tokens and related connection pools.

Expired tokens are no longer immediately deleted so that its possible for them to be listed by "tokens" for up to two hours.

Prevent a lock inversion introduced in v0.208 that can result in a deadlock involving the GLOCK and the rx call.lock. The deadlock can occur if a cell's list of location servers expires and during the rebuild an rx abort is issued.

Add support for rxkad "auth" mode rx connections in addition to "clear" and "crypt". "auth" mode provides integrity protection without privacy.

Add support for yfs-rxgk "clear" and "auth" rx connection modes.

Do not leak a directory buffer page reference when populating a directory page fails.

Re-initialize state when populating a disk cache entry using the fast path fails and a retry is performed using the slow path. If the data version changes between the attempts it is possible for truncated disk cache data to be treated as valid.

Log warnings if a directory lookup operation fails with an EIO error. An EIO error indicates that an invalid directory header, page header, or directory entry was found.

Do not overwrite RX errors with local errors during Direct-I/O and StoreMini operations. Doing so can result in loss of VBUSY, VOFFLINE, UAENOSPC, and similar errors.

Correct a direct i/o code path which could overwrite a fileserver returned error code with a local RXGEN_CC_UNMARSHAL error.

Correct the StoreMini code path which could overwrite a fileserver returned error code with a local RXGEN_CC_UNMARSHAL error.

Ensure the rx call object is not locked when writing to the network socket.

Removed all knowledge of the KERNEL global lock from RX. Acquiring the GLOCK from RX is never safe if any other lock is held. Doing so is a lock order violation that can result in deadlocks.

Fixed a race in the opr_reservation system that could produce a cache entry reference undercount.

If a directory hash chain contains a circular link, a buffer page reference could be leaked for each traversal.

Each AFS3 directory header and page header contains a magic tag value that can be used in a consistency check but was not previously checked before use of each header. If the header memory is zero filled during a lookup, the search would fail producing an ENOENT error. Starting with this release the magic tag values are validated on each use. An EIO error is returned if there is a tag mismatch.

"fs setcrypt -crypt auth" is now a permitted value. The "auth" mode provides integrity protection but no privacy protection.

Add new "aklog -levels " option which permits requesting "clear" and "auth" modes for use with yfs-rxgk.

Update MKShim to Apple OpenSource MITKerberosShim-79.

Report KLL errors via a notification instead of throwing an exception which (if not caught) will result in process termination.

If an exception occurs while executing "unlog" catch it and ignore it. Otherwise, the process will terminate.

New to v2021.04 (22 April 2021)

Primarily bug fixes for issues that have been present for years.

A possibility of an infinite kernel loop if a rare file write / truncate pattern occurs.

A bug in silly rename handling that can prevent cache manager initiated garbage collection of vnodes.

New to v0.209 (13 March 2021)

fs setserverprefs and fs getserverprefs updated to support IPv6 and CIDR specifications.

Improved error handling during fetch data and store data operations.

Prevents a race between two vfs operations on the same directory which can result in caching of out of date directory contents.

Use cached mount point target information instead of evaluating the mount point's target upon each access.

Avoid rare data cache thrashing condition.

Prevent infinite loop if a disk cache error occurs after the first page in a chunk is written.

Network errors are supposed to be returned to userspace as ETIMEDOUT. Previously some were returned as EIO.

When authentication tokens expire, reissue the fileserver request anonymously. If the anonymous user does not have permission either EACCES or EPERM will be returned as the error to userspace. Previously the vfs request would fail with an RXKADEXPIRED or RXGKEXPIRED error.

If growth of an existing connection vector fails, wait on a call slot in a previously created connection instead of failing the vfs request.

Volume and fileserver location query infrastructure has been replaced with a new modern implementation.

Replace the cache manager's token management infrastructure with a new modern implementation.

New to v0.206 (12 January 2021) - Bug fixes

Prevents a possible panic during unmount of /afs.

Improved failover and retry logic for offline volumes.

New to v0.205 (24 December 2020) - Bug fixes

Volume name-to-id cache improvements

Fix expiration of name-to-id cache entries

Control volume name-to-id via sysctl

Query volume name-to-id statistics via sysctl

Improve error handling for offline volumes

Fix installer to prevent unnecessary installation of Rosetta 2 on Apple Silicon

New to v0.204 (25 November 2020) - Bug fix for macOS Big Sur

v0.204 prevents a kernel panic on Big Sur when AuriStorFS is stopped and restarted without an operating system reboot.

introduces a volume name-to-id cache independent of the volume location cache.

New to v0.203 (13 November 2020) - Bug fix for macOS

v0.203 prevents a potential kernel panic due to network error.

New to v0.201 (12 November 2020) - Universal Big Sur (11.0) release for Apple Silicon and Intel

v0.201 introduces a new cache manager architecture on all macOS versions except for High Sierra (10.12). The new architecture includes a redesign of:

kernel extension load

kernel extension unload (not available on Big Sur)

/afs mount

/afs unmount

userspace networking

The conversion to userspace networking will have two user visible impacts for end users:
The Apple Firewall as configured by System Preferences -> Security & Privacy -> Firewall is now enforced. The "Automatically allow downloaded signed software to receive incoming connections" includes AuriStorFS.

Observed network throughput is likely to vary compared to previous releases.

On Catalina the "Legacy Kernel Extension" warnings that were displayed after boot with previous releases of AuriStorFS are no longer presented with v0.201.

AuriStorFS /afs access is expected to continue to function when upgrading from Mojave or Catalina to Big Sur. However, as AuriStorFS is built specifically for each macOS release, it is recommended that end users install a Big Sur specific AuriStorFS package.
AuriStorFS on Apple Silicon supports hardware accelerated aes256-cts-hmac-sha1-96 and aes128-cts-hmac-sha1-96 using AuriStor's proprietary implementation.

New to v0.200 (4 November 2020) - Final release for macOS El Capitan (10.11)

The network path between a client and a server often traverses one or more network segments separated by NAT/PAT devices. If a NAT/PAT times out a RPCs endpoint translation mid-call, this can result in an extended delay before failure and the server being marked down, or worse, a call that never terminates and a client that appears to hang until the fileserver is restarted.

This release includes significant changes to the RX stack and the UNIX cache manager to detect such conditions, fail the calls quickly and detect when it is safe to retry the RPC.

NAT/PAT devices that drop endpoint mappings while in use are anti-social and can result in unwanted delays and even data loss. Whenever possible they should be avoided when they can be. That said, the changes in this release are a huge step to making the loss of endpoint mappings tolerable.

Fix segmentation fault of Backgrounder when krb5_get_credentials() fails due to lack of network connectivity.

Fix the "afsd" rxbind option which was ignored if the default port, 7001, is in use by another process on the system.

If a direct i/o StoreData or FetchData RPC failed such that it must be retried, the retried RPC would fail due to an attempt to Fetch or Store the wrong amount of data. This is fixed.

Servers are no longer marked down if RPCs fail with RX_CALL_PEER_RESET, RX_CALL_EXCEEDS_WINDOW, or RX_PROTOCOL_ERROR. RPCs that are safe to retry are retried.

Fixed a race between a call entering error state and call completion that can result in a the call remaining in the DALLY state and the connection channel remaining in use. If this occurs during process or system shutdown it can result in a deadlock.

During shutdown cancel any pending delayed aborts to prevent a potential deadlock. If a deadlock occurs when unloading a kernel module a reboot will be required.

Updated cellservdb.conf

New to v0.197.1 (31 August 2020) and v0.198 (10 October 2020)

Prevent Dead vnode has core/unlinkedel/flock panic introduced in v0.197.

New to v0.197 (26 August 2020)

A new callback management framework for UNIX cache managers reduces the expense of processing volume callback RPCs from O(number of vcache objects) to O(1). A significant amount of lock contention has been avoided. The new design reduces the risk of the single callback service worker thread blocking. Delays in processing callbacks on a client can adversely impact fileserver performance and other clients in the cell.

Bulk fetch status RPCs are available on macOS for the first time. Bulk fetch status permits optimistic caching of vnode status information without additional round-trips. Individual fetch status RPCs are no longer issued if a bulk status fails to obtain the required status information.

Hardware accelerated crypto is now available for macOS cache managers. AuriStor's proprietary aes256-cts-hmac-sha1-96 and aes256-cts-hmac-sha512-384 implementations leverage Intel processor extensions: AESNI AVX2 AVX SSE41 SSSE3 to achieve the fastest encrypt, decrypt, sign and verify times for RX packets.

This release optimizes the removal of "._" files that are used to store extended attributes by avoiding unnecessary status fetches when the directory entry is going to be removed.

When removing the final directory entry for an in-use vnode, the directory entry must be silly renamed on the fileserver to prevent removal of the backing vnode. The prior implementation risked blindly renaming over an existing silly rename directory entry.

Behavior change! When the vfs performs a lookup on ".", immediately return the current vnode.

if the object is a mount point, do not perform fakestat and attempt to resolve the target volume root vnode.

do not perform any additional access checks on the vnode. If the caller already knows the vnode the access checks were performed earlier. If the access rights have changed, they will be enforced when the vnode is used just as they would have if the lookup of "." was performed within the vfs.

do not perform a fetch status or fetch data rpcs. Again, the same as if the lookup of "." was performed within the vfs.

Volumes mounted at more than one location in the /afs namespace are problematic on more than one operating system that do not expect directories to have more than one parent. It is particularly problematic if a volume is mounted within itself. Starting with this release any attempt to traverse a mountpoint to the volume containing the mountpoint will fail with ENODEV.

When evaluating volume root vnodes, ensure that the vnode's parent is set to the parent directory of the traversed mountpoint and not the mountpoint. Vnodes without a parent can cause spurious ENOENT errors on Mojave and later.

v0.196 was not publicly released.

New to v0.195 (14 May 2020)

This is a CRITICAL update for AuriStorFS macOS clients.

In Sep 2019 AuriStorFS v0.189 was released which provided faster and less CPU intensive writing of (>64GB) large files to /afs. These improvements introduced a hash collision bug in the store data path of the UNIX cache manager which can result in file corruption. If a hash collision occurs between two or more files that are actively being written to via cached I/O (not direct I/O), dirty data can be discarded from the auristorfs cache before it is written to the fileserver creating a file with a range of zeros (a hole) on the fileserver. This hole might not be visible to the application that wrote the data because the lost data was cached by the operating system. This bug has been fixed in v0.195 and it is for this reason that v0.195 has been designated a CRITICAL release for UNIX/Linux clients.

While debugging a Linux SIGBUS issue, it was observed that receipt of an ICMP network error in response to a transmitted packet could result in termination of an unrelated rx call and could mark a server down. If the terminated call is a StoreData RPC, permanent data loss will occur. All Linux clients derived from the IBM AFS code base experience this bug. The v0.195 release prevents this behavior.

This release includes changes that impact all supported UNIX/Linux cache managers. On macOS there is reduced lock contention between kernel threads when the vcache limit has been reached.

The directory name lookup cache (DNLC) implementation was replaced. The new implementation avoids the use of vcache pointers which did not have associated reference counts, and eliminates the invalidation overhead during callback processing. The DNLC now supports arbitrary directory name lengths; the prior implementation only cached entries with names not exceeding 31 characters.

Prevent matching arbitrary cell name prefixes as aliases. For example "/afs/y" should not be an alias for "your-file-system.com". Some shells, for example "zsh", query the filesystem for names as users type. Delays between typed characters result in filesystem lookups. When this occurs in the /afs dynroot directory, this could result in cellname prefix string matches and the dynamic creation of directory entries for those prefixes.

New to v0.194 (2 April 2020)

This is a CRITICAL release for all macOS users. All prior macOS clients whether AuriStorFS or OpenAFS included a bug that could result in data corruption either when reading or writing.

This release also fixes these other issues:

sign and notarize installer plugin "afscell" bundle. The lack of digital signature prevented the installer from prompting for a cellname on some macOS versions.

prevent potential for corruption when caching locally modified directories.

v0.193 was withdrawn due to a newly introduced bug that could result in data corruption.

New to v0.192 (30 January 2020)

The changes improve stability, efficiency, and scalability. Post-0.189 changes exposed race conditions and reference count errors which can lead to a system panic or deadlock. In addition to addressing these deficiencies this release removes bottlenecks that restricted the number of simultaneous vfs operations that could be processed by the AuriStorFS cache manager. The changes in this release have been successfully tested with greater than 400 simultaneous requests sustained for for several days.

New to v0.191 (16 December 2019)

Restore keyed cache manager capability broken in v0.189.

Add kernel module version string to AuriStorFS Preference Pane.

Other kernel module bug fixes.

New to v0.190 (14 November 2019)

Short-circuit busy volume retries after volume or volume location entry is removed.

New to v0.189 (28 October 2019)

Faster "git status" operation on repositories stored in /afs.

Faster and less CPU intensive writing of (>64GB) large files to /afs. Prior to this release writing files larger than 1TB might not complete. With this release store data throughput is consistent regardless of file size. (See "UNIX Cache Manager large file performance improvements" later in this file).

macOS Catalina (8 October 2019)

AuriStorFS v0.188 released for macOS Catalina (10.15)

New to v0.188 (23 June 2019)

Increased clock resolution for timed waits from 1s to 1ns

Added error handling for rx multi rpcs interrupted by signals

New to v0.186 (29 May 2019)

v0.184 moved the /etc/yfs/cmstate.dat file to /var/yfs. With this change afsd would fail to start if /etc/yfs/cmstate.dat exists but contains invalid state information. This is fixed.

v0.184 introduced a potential deadlock during directory processing. This is fixed.

Handle common error table errors obtained outside an afs_Analyze loop. Map VL errors to ENODEV and RX, RXKAD, RXGK errors to ETIMEDOUT

Log all server down and server up events. Transition events from server probes failed to log messages.

RX RPC networking:

If the RPC initiator successfully completes a call without consuming all of the response data fail the call by sending an RX_PROTOCOL_ERROR ABORT to the acceptor and returning a new error, RX_CALL_PREMATURE_END, to the initiator.
Prior to this change failure to consume all of the response data would be silently ignored by the initiator and the acceptor might resend the unconsumed data until any idle timeout expired. The default idle timeout is 60 seconds.

Avoid transmitting ABORT, CHALLENGE, and RESPONSE packets with an uninitialized sequence number. The sequence number is ignored for these packets but set it to zero.

New to v0.184 (26 March 2019)

The initial congestion window has been reduced from 10 Rx packets to 4. Packet reordering and loss has been observed when sending 10 Rx packets via sendmmsg() in a single burst. The lack of udp packet pacing can also increase the likelihood of transmission stalls due to ack clock variation.

The UNIX Cache Manager underwent major revisions to improve the end user experience by revealing more error codes, improving directory cache efficiency, and overall resiliency. The cache manager implementation was redesigned to be more compatible with operating systems such as Linux and macOS that support restartable system calls. With these changes errors such as "Operation not permitted", "No space left on device", "Quota exceeded", and "Interrupted system call" can be reliably reported to applications. Previously such errors might have been converted to "I/O error".

New to v0.180 (9 November 2018)

RX reliability and performance improvements for high latency and/or lossy network paths such as public wide area networks.

A fix for a macOS firewall triggered kernel panic introduced in v0.177.

New to v0.177 (17 October 2018)

A fix to AuriStor's RX implementation bug introduced in v0.176 that interferes with communication with OpenAFS and IBM Location and File Services.

New to v0.176 (3 October 2018)

AuriStor's RX implementation has undergone a major upgrade of its flow control model. Prior implementations were based on TCP Reno Congestion Control as documented in RFC5681; and SACK behavior that was loosely modelled on RFC2018. The new RX state machine implements SACK based loss recovery as documented in RFC6675, with elements of New Reno from RFC5682 on top of TCP-style congestion control elements as documented in RFC5681. The new RX also implements RFC2861 style congestion window validation.

When sending data the RX peer implementing these changes will be more likely to sustain the maximum available throughput while at the same time improving fairness towards competing network data flows. The improved estimation of available pipe capacity permits an increase in the default maximum window size from 60 packets (84.6 KB) to 128 packets (180.5 KB). The larger window size increases the per call theoretical maximum throughput on a 1ms RTT link from 693 mbit/sec to 1478 mbit/sec and on a 30ms RTT link from 23.1 mbit/sec to 49.39 mbit/sec.

Improve shutdown performance by refusing to give up callbacks to known unreachable file servers and apply a shorter timeout period for the rest.

Permit RXAFSCB_WhoAreYou to be successfully executed after an IBM AFS or OpenAFS fileserver unintentionally requests an RX service upgrade from RXAFSCB to RXYFSCB.

RXAFS timestamps are conveyed in unsigned 32-bit integers with a valid range of 1 Jan 1970 (Unix Epoch) through 07 Feb 2106. UNIX kernel timestamps are stored in 32-bit signed integers with a valid range of 13 Dec 1901 through 19 Jan 2038. This discrepency causes RXAFS timestamps within the 2038-2106 range to display as pre-Epoch.

RX Connection lifecycle management was susceptible to a number of race conditions that could result in assertion failures, the lack of a NAT ping connection to each file server, and the potential reuse of RX connections that should have been discarded.

This release includes a redesigned lifecycle that is thread safe, avoids assertions, prevents NAT ping connection loss, and ensures that discarded connections are not reused.

The 0.174 release unintentionally altered the data structure returned to xstat_cm queries. This release restores the correct wire format.

Since v0.171, if a FetchData RPC fails with a VBUSY error and there is only one reachable fileserver hosting the volume, then the VFS request will immediately with an ETIMEDOUT error ("Connection timed out").

v0.176 corrects three bugs that contributed to this failure condition. One was introduced in v0.171, another in 0.162 and the final one dates to IBM AFS 3.5p1.

The intended behavior is that a cache manager, when all volume sites fail an RPC with a VBUSY error, will sleep for up to 15 seconds and then retry the RPC as if the VBUSY error had never been received. If the RPC continues to receive VBUSY errors from all sites after 100 cycles, the request will be failed with EWOULDBLOCK ("Operation would block") and not ETIMEDOUT.

Prefer VOLMISSING and VOLBUSY error states to network error states when generating error codes to return to the VFS layer. This will result in ENODEV ("No such device") errors when all volume sites return VNOVOL or VOFFLINE errors and EWOULDBLOCK ("Operation would block") errors when all volume sites return VBUSY errors. (v0.176)

New to v0.174 (24 September 2018)

macOS Mojave (10.14) support

New to v0.170 (27 April 2018)

Faster processing of cell configuration information by caching service name to port information.

RX call sequence number rollover to permit calls that require the transmission of more than 5.5TB of data.

Command parser Daylight Saving Time bug fix

Fix a bug that prevented immediate access to a mount point created with "fs mkmount" on the same machine.

Fix the setting of "[afsd] sysnames = " during cache manager startup.

New to v0.168 (6 March 2018)

Corrects "fs setacl -negative" processing [CVE-2018-7168]

Improved reliability for keyed cache managers. More persistent key acquisition renewals.

Major refresh to cellservdb.conf contents.

DNS SRV and DNS AFSDB records now take precedence when use_dns = yes

Kerberos realm hinting provided by
kerberos_realm = [REALM]

DNS host names are resolved instead of reliance on hard coded IP addresses
The cache manager now defaults to sparse dynamic root behavior. Only thiscell and those cells that are assigned aliases are included in /afs directory enumeration at startup. Other cells will be dynamically added upon first access.
Several other quality control improvements.

New to v0.167 (7 December 2017)

Addresses a critical remote denial of service vulnerability [CVE-2017-17432]
Alters the volume location information expiration policy to reduce the risk of single points of failures after volume release operations.
'fs setquota' when issued with quota values larger than 2TB will fail against OpenAFS and IBM AFS file servers
Memory management improvements for the memory caches.

New to v0.164 (11 November 2017)

Internal cache manager redesign. No new functionality.

New to v0.160 (21 September 2017)

Support for OSX High Sierra's new Apple File System (APFS). Customers must upgrade to v0.160 or later before upgrading to OSX High Sierra.
Reduced memory requirements for rx listener thread
Avoid triggering a system panic if an AFS local disk cache file is deleted or becomes inaccessible.
Fixes to "fs" command line output

New to v0.159 (7 August 2017)

Improved failover behavior during volume maintenance operations
Corrected a race that could lead the rx listener thread to enter an infinite loop and cease processing incoming packets.

New to v0.157 (12 July 2017)

Bundled with Heimdal 7.4 to address CVE-2017-11103 (Orpheus' Lyre puts Kerberos to sleep!)
"vos" support for volume quotas larger than 2TB.
"fs flushvolume" works
Fixed a bug that can result in a system panic during server capability testing

New to v0.150

AuriStorFS file server detection improvements

New to v0.149

rxkad encryption is enabled by default. Use "fs setcrypt off" to disable encryption when tokens are available.
Fix a bug in atomic operations on Sierra and El Capitan which could adversely impact Rx behavior.

New to v0.128

Extended attribute ._ files are automatically removed when the associated files are unlinked
Throughput improvements when sending data

New to v0.121

OSX Sierra support

New to v0.117

Cache file moved to a persistent location on local disk
AuriStor File System graphics
Improvements in Background token fetch functionality
Fixed a bug introduced in v0.44 that could result in an operating system crash when enumerating AFS directories containing Unicode file names (v0.106)
El Capitan security changes prevented Finder from deleting files and directories. As of v0.106, the AuriStor OSX client implements the required functionality to permit the DesktopHelperService to securely access the AFS cache as the user permitting Finder to delete files and directories.

Features:

Not vulnerable to OPENAFS-SA-2015-007.
Office 2011 can save to /afs.
Office 2016 can now save files to /afs.
OSX Finder and Preview can open executable documents without triggering a "Corrupted File" warning. .AI, .PDF, .TIFF, .JPG, .DOCX, .XLSX, .PPTX, and other structured documents that might contain scripts were impacted.
All file names are now stored to the file server using Unicode UTF-8 Normalization Form C which is compatible with Microsoft Windows.
All file names are converted to Unicode UTF-8 Normalization Form D for processing by OSX applications.

Known issues:

None

macOS Installer (10.9 Mavericks)

Release Notes

Known Issues

If the Kerberos default realm is not configured, a delay of 6m 59s can occur before the AuriStorFS Backgrounder will acquire tokens and display its icon in the macOS menu. This is the result of macOS performing a Bonjour (MDNS) query in an attempt to discover the local realm.
"aklog" will fail to acquire tokens if the current working directory is located in an inaccessible AFS volume. When the "cwd" is inaccessible, the Heimdal framework is unable to initialize the Kerberos v5 context and reports an out of memory error.

New v2021.05-64 (22 Aug 2025)

The v2021.05-64 release fixes a race condition which can lead to a kernel panic when executing the "unlog" command.

v2021.05-63 (31 May 2025)

When AuriStorFS cache managers added support for per-file ACLs in v0.1, the masking of access rights based upon the inode's S_IRWXU mode bits were skipped when reading objects from AuriStorFS fileservers. The masking continued to be applied when reading objects from OpenAFS fileservers. This release applies S_IRWXU mode bits masking of granted access rights to both AuriStorFS and OpenAFS fileservers.
The existing fake stat mountpoint evaluation logic assumes that the fileserver returned a AFSFetchStatus.ParentVnode that refers to the parent directory from which the mountpoint was located. However, this is only guaranteed to be true if the mountpoint does not have a per-file ACL (which is permitted) and has never been cross-directory hard linked (which is also permitted).
This release improves the error handling logic to fail the fake stat evaluation if the AFSFetchStatus.ParentVnode does not match expectations.
This release fixes a metadata inconsistency when a mountpoint is created when disconnected mode is enabled. A mountpoint should be reported to the vfs as a directory and not a symlink.
The cache manager must track for each inode whether the object's status infomration was received from an AFS3 fileserver or an AuriStorFS fileserver. This release alters when that information is recorded to ensure that it is recorded each time the inode's status information is updated by a fileserver response.
This knowledge is required to properly determine how to interpret the status information.
Directory entries referring to mountpoints and volume roots must not be stored within the Directory Name Lookup Cache (DNLC). Instead, full lookups must be performed each time to ensure proper mountpoint resolution.
Prior to this release locally created mountpoints were stored into the DNLC by mistake.

v2021.05-62 (24 April 2025)

macOS Sequoia v15.4 vs Cisco AnyConnect VPN

Beginning with macOS 15.4, use of Cisco AnyConnect VPN breaks DNS SRV, AFSDB and TXT record lookups performed using the cross-platform DNS resolver API. As a result, clients which rely upon DNS-based zero-configuration resolution of Kerberos realms and/or AFS cells database services are unable to access those resources. This release switches the macOS builds to use the Apple DNS API.
The lookup failures impacted all command line tools, clients and services.
Note: Communication failures can occur if the VPN is configured by the organization to block the transmission of fragmented UDP packets. Set [afsd] rxmaxmtu = 1328 to ensure all RX packets are small enough to traverse the VPN link.
This release improves consistency of vnode metadata interpretation when the status information is received via an RPC which does not return a callback.

v2021.05-61 (31 March 2025)

Instead of allocating the same sized hash table for the vcache and dcache objects regardless of how many elements are expected to be stored, scale the hash table sizes. The vcache hash table will scale from 4096 buckets to 131072 buckets with a target of 4 entries per bucket. The dcache hash tables will scale from 4096 buckets to 65536 buckets with a target of 2 entries per bucket.
Historically, afs cache managers which create symlinks (including mountpoints) locally allocate a fake infinite lifetime callback to permit applications to use the symlink (or mountpoint) without requiring FetchStatus RPCs to be issued to the fileserver. This approach has drawbacks because it means that fileservers will not issue a callback break when the volume containing the symlink is taken offline, moved, deleted, etc. It also means that a callback will not be sent if another cache manager removes the symlink.
Beginning with this release the AuriStorFS cache manager will no longer allocate a fake infinite lifetime callback and will instead rely upon fileserver allocated callbacks.
"fs makemount" is supposed to create a mountpoint even if the target volume does not exist or is inaccessible. v2021.05 introduced support for creating mountpoints which kafs which broke the ability to create mountpoints to non-existent volumes.

v2021.05-60 (6 March 2025)

The cache manager retries fileserver RPCs up to 100 times when the fileserver replies with VBUSY, VOFFLINE or VSALVAGE. The cache manager fails the application request after the limit is reached. Prior to this release, the failure would result in the cache manager returning an ENODEV error to the application. But ENODEV indicates that the required volume does not exist and the fileserver has not indicated that the volume does not exist. The cache manager will now return EWOULDBLOCK to the application because the volume might be temporarily inaccessible.
Many OpenAFS and AuriStorFS fileservers can return VNOVOL as a side effect of a race between a volume going offline so that control can be given to the Volume Management Service (volserver) and an incoming RPC requesting that the volume be attached. A VNOVOL can also be returned because the volume was temporarily deleted from the vice partition prior to being replaced by a new clone, or prior to the fileserver marking the volume group as having been moved to another fileserver.
The fileserver replying to an RPC with VNOVOL cannot be trusted to mean that the volume has been permanently removed unless it is matched by the Location Service reporting the Volume Location (VL) entry has been deleted. From now on when a VNOVOL response is received and the VL entry exists, the cache manager will return EWOULDBLOCK instead of ENODEV to the application.
Prior to this release, any VL error code returned from the Location Service would be interpreted as meaning that the requested VL entry does not exist. Starting with this release only the VL_NOENT error will be interpreted as the VL entry does not exist.
Restrict the internal use of the ENOENT error code so the only use of ENOENT is to indicate that a directory entry does not exist. Previously, ENOENT could be returned if a disk cache entry could not be allocated or found. A disk cache related ENOENT returned when processing a directory might have been confused for the requested directory entry not existing.
Restrict the internal use of the ENODEV error code so that the only meaning of ENODEV is that the required volume does not exist. Previously, ENODEV could be returned due to transient errors when failing to contact the Location Service.

v2021.05-59 (13 February 2025)

The fix for garbage collection of idle unixuser objects introduced in v2021.05-56 was incomplete. As a result, Rx client connections might not be garbage collected.

v2021.05-57 (9 February 2025)

No new changes for macOS.

v2021.05-56 (7 February 2025)

Minor bug fixes to the Rx RPC implementation
The fix for garbage collection of idle unixuser objects introduced in v2021.05-55 was incomplete. As a result, Rx client connections might not be garbage collected.

v2021.05-55 (23 January 2025)

Minor bug fixes to the Rx RPC implementation
During garbage collection of idle unixuser objects, any Rx client connections associated with the unixuser (uid or pag) that were in use by another thread would not have their cache manager reference dropped. As a result the Rx client connections could never be garbage collected which is problematic on systems which churn through many unixusers. The behavior has been changed so that unixusers objects with in use Rx connections will not be freed. This permits future attempts to garbage collect unixusers to do so only after the Rx connections can also be freed.
Although this misbehavior has been present for decades, it is only since v2021.05-52 that the problem has been observed. It is possible that the changes to reduce the holding of the GLOCK has created a window of opportunity for connections to be simultaneously in use during unixuser garbage collection.

v2021.05-54 (19 January 2025)

The UNIX cache manager changes continue AuriStor's efforts to address a report that was received at the end of September. The end user site uses an OpenAFS cell to distribute software for execution by IBM Spectrum LSF (load sharing facility) managed compute nodes. Each of the approximately one dozen compute nodes is configured with approximately 100 processor cores and 1TB of ram. Each job scheduled by LSF runs in a newly allocated Process Authentication Group (PAG). Some jobs run for hours or days; others complete in a few seconds. LSF has been observed scheduling more than 3000 jobs in under a minute with peak counts of cache manager user objects at 1300. Everything ran well until it didn't. After some unexplained trigger event, all of the remaining free memory on the system would be consumed in about an hour causing the system to panic.

Over the last five months AuriStor has diligently worked to develop theories, optimize the AuriStorFS UNIX cache manager to address each theory; then move on to subsequent ideas once prior theories were disproved. Over this time we have replaced the memory management lifecycle for access rights entries and rx events. We have made it possible for Linux permission checks to be processed without acquiring the GLOCK in the majority of access patterns thereby increasing parallelism. The lifecycle management of PAG allocation and user entity tracking has been rewritten to avoid the GLOCK. The Linux inode lifecycle management has been rewritten which some sites attribute to elimination of swapping on their systems. All of these changes increased throughput and parallelism within and without the cache manager. However, none of them prevented the memory consumption once the unknown trigger event occurred.

It is with great pleasure that we can report that the trigger event and the source of the memory consumption have been identified. The trigger is a damaged Rx packet header which causes the AuriStor Rx stack to place an Rx connection into an RX_PROTOCOL_ERROR error state when there is an in-flight call to an OpenAFS fileserver. The memory consumption is due to the infinite loop that occurs when an Rx connection is in an error state and the error code is not one of the error codes which forces replacement of the Rx connection. Within each pass of the loop a structure is allocated to track the history of per-server errors so they can be logged when hard mount mode is enabled for the cell. AuriStor Rx permits up to 12 calls on each Rx connection. As a result there can be up to a dozen processes each spinning and allocating memory when the Rx connection is placed into the error state.

Days before Christmas 2023 AuriStor identified a race condition which can result in damaged Rx DATA packets being written to the network. Initial transmission of Rx DATA packets can be sent either from an application thread or an Rx listener. Packet retransmission occurs either in an Rx listener thread or in the Rx event thread. The race permits a second thread to modify the Rx packet headers while the sendmsg() syscall from the original transmission is executing. This race is possible because the original thread might exhaust its timeslice during the sendmsg() call and it might be many milliseconds before it is rescheduled on a busy system. In the meantime, packet retransmissions might be initiated either by an Rx listener thread processing an ACK packet or the Rx event thread processing a timeout event. If the initial packet transmission completes while the Rx headers are being rewritten the resulting packet sent on the wire might be garbage. AuriStor observed that if the modified packets were jumbograms that silent data corruption could occur so we disabled all use of jumbograms in v2021.05-35 We also notified both IBM and OpenAFS of the problem.

v2021.05-53 modified the RPC result analyze logic to force the replacement of Rx connections in an error state except when the error signals a token expiration. When a token is expired there is no benefit to replacing the connection until new tokens have been acquired. This release further ensures that infinite memory allocation cannot occur by tracking only the most recent error per fileserver; and only does so if hard mount mode is enabled for the cell.

Processes that issued syscalls which became stuck in a RPC retry loop could not be interrupted. In addition to the unintentional retry loops there are intentional retry loops such as those that occur because a fileserver repeated fails the call with a “VBUSY” error. “VBUSY” originally meant that the required volume was temporarily in use by the volume management service (“volserver”) and would be back in service soon so please wait. Beginning with IBM AFS 3.4 the error code was used to force a retry for reasons having nothing to do with the requested volume’s availability. IBM AFS 3.4 introduced the concept of multi-homed clients which meant that the fileserver could no longer track the identity of a cache manager by its IPv4 address. Instead, each cache manager randomly generates a UUID which the fileserver must query each time an incoming RPC is received on a previously unrecognized Rx connection. An Rx connection might be unrecognized because it’s new, or because the client’s IP address changed, or because the fileserver garbage collected its state information. Regardless of the reason, if a call is received and there is no Rx connection binding to a cache manager UUID, then the fileserver application worker thread to which the call was assigned must issue an outgoing RPC to the client’s RXAFSCB service. Each client has at most one worker thread. Prior to this release, every incoming RXAFSCB call required acquisition of the GLOCK before it could execute. If the GLOCK is unavailable, the RXAFSCB thread blocks and no other pending RXAFSCB calls can be processed. If a RXAFSCB call blocks for too long, the fileserver timeouts trigger and the fileserver replies to the incoming RXAFS call with “VBUSY”. “VBUSY” can also be sent if the fileserver’s quota on the number of calls from the same endpoint is exceeded.

The v2021.05-54 release removes the GLOCK requirement for many RXAFSCB RPCs including all of the UUID query RPCs and all whole volume callback RPCs. Callback RPCs for individual vnodes must still acquire the GLOCK. However, it is hoped that these changes will significantly reduce the time required to process the queries generated due to unrecognized Rx connections. The AuriStor Rx stack permits a dozen calls to be issued on a single logical connection which the fileserver sees as three unique Rx connections from the same endpoint.

Each Unix UID or PAG ID must use an independent Rx connection to each fileserver. On a system which processes /afs syscalls for 1200 UID or PAG IDs in a minute, there is the potential for 3600 Rx connections per fileserver. When the fileserver is OpenAFS, this quickly triggers per endpoint throttling and generates VBUSY storms which can result in unlucky client processes experiencing starvation. With this release, starving processes can be interrupted and RXAFSCB UUID queries can be completed without blocking.

AuriStor believes this release will provide significant benefits for large scale compute nodes, web servers and time sharing systems.

v2021.05-53 (4 January 2025)

v2021.05-52 switched the memory allocation strategy for rx_event objects from a global free list to reliance upon the operating system's memory allocator to allocate individual objects which are freed upon rx_event completion or cancelation. Reliance upon the system allocator introduces the risk of allocation failures due to out of memory conditions. If allocation of an rx_event object fails, a reference count leak on the associated rx_connection or rx_call object can occur preventing garbage collection of the object.
This release handles rx_event allocation failures:
- delayed ACK or delayed ABORT events are replaced by immediate transmission of the ACK or ABORT
- other cases fail the call or connection with an RX_OUT_OF_MEMORY error which is translated on the wire to RX_WIRE_CALL_DEAD (-1).
After executing a call to a fileserver, if the Rx connection is in an error state for any reason other than expired tokens, then schedule the Rx connection to be replaced the next time a call is attempted. All calls issued on an Rx connection that is already in an error state fail immediately preventing further communication with the associated fileserver from the UID/PAG.
Rx connections in an expired token error state are not replaced until new tokens are set.
Rx connections can be placed into an error state either due to local or remote failures. Such failures are often generated by the in-use security class (RXKAD/RXGK) but might be due to an out-of-memory condition, system clock advancement or reversal, or even application errors.
This release includes internal code changes to the management of servers, connections and access rights which will be used by a future release to separate PAG ID allocation from the system's UID namespace.
If 'afsd' is configured with a keytab, receipt of a PoweredOn event will force an attempt to acquire system tokens.
On supported macOS versions, enforce the use of code signing identities to authenticate mach messages involving the AFSBackgrounder and the AFSPreference panel.

v2021.05-52 (5 December 2024)

The v2021.05-50 replacement of the linked list implementation of the vcache access cache with a btree and last lookup cache introduced a memory corruption bug which is fixed in this release.
Eliminate the GLOCK requirement for the collection of function call and RPC statistics collection.
Replace small space allocation with a GLOCK free allocator.
Reimplement the cache manager tracing logging routines so they no longer require the GLOCK.
The Rx networking stack schedules many different types of time sensitive events associated with individual packet retransmission, delayed ack and abort processing, path mtu detection, keep alives, and other categories. Prior to this release, all memory allocated to Rx events objects would not be freed to the operating system until the userspace process terminated or the kernel module was unloaded.
Beginning with this release, instead of having a single, global, free list for rxevents, the system allocator is trusted to be able to allocate memory quickly. This simplifies the Rx implementation, and should be more efficient, as most system allocators will have per thread or core slabs, removing the contention of the global lock protecting the rxevent free list. In addition, the change protects against a network event allocating memory that cannot be readily freed.
On non-Total Store Ordering (TSO)architectures, relaxed atomics used as reference counters do not ensure completion of all prior stores or recency of later loads. This release replaces all read-modify-write atomics to be acquire-release, which, among other things, ensures that only the intended ordering is possible.

v2021.05-51 (26 November 2024)

Fix a vcache (aka inode) reference leak that can occur during an access rights query for a non-directory object if the neither the calling process nor the cache manager possesses tokens for the cell containing the object. The leak was introduced in v2021.05-45.
When scanning the token list for each user searching for expired tokens the user spinlock must be held to prevent races. The updated user management implementation pulled into v2021.05-45 failed to hold the spinlock.
During cache manager module shutdown reset global structures to the module initialization state. Doing so improves the likelihood that the module can be reinitialized.

v2021.05-50 (23 November 2024)

The linked list implementation of the vcache access rights cache was replaced with a btree and a last lookup cache. The btree design is more efficient for lookups and removals, and scales better as the number of uids and/or process authentication groups (pags) accessing /afs increases.
Prune access rights for expired users. Doing so reduces the cost of subsequent lookups.
Cache the access rights obtained for a uid|pag via the "anyAccess" field. Previously, a uid|pag that obtained its rights from the "anyAccess" field had to perform and fail a rights lookup on each and every vcache access.
When setting or clearing tokens for a single cell, no longer invalidate the access rights cached for the uid|pag for all cells. Instead, only invalidate the access rights cached for the cell whose tokens were set or cleared.

v2021.05-49 (16 November 2024)

The output of "tokens" command failed to report yfs-rxgk tokens was broken starting in v2021.05-46

v2021.05-48 (12 November 2024)

Preallocated buffer overflows in XDR responses (CVE-2024-10397)
The AuriStorFS and AFS3 RPC suites rely upon Sun RPC XDR to marshal binary data structures for network transfer. The AuriStor XDR implementation is derived from Sun Microsystems' Sun RPC code base. The Sun RPC XDR API permits memory for output parameters to (optionally) be preallocated which can result in various classes of memory corruption and/or memory leaks in RPC initiator processes.
The AuriStorFS v2021.05-48 release introduces additional data length validation checks within the AuriStor XDR implementation and prohibits the use of preallocated memory for string output parameters or fields. All cache managers, servers and command line tools are modified by these changes.

v2021.05-46 (28 October 2024)

Cache Manager:
- Prevent a kernel memory leak when server preferences are set via the yfs-client.conf [afsd] configuration or via "fs setserverprefs".
- Directory enumeration of a truncated directory now returns an error instead of assuming the end of the directory has been reached.
- Since AFS 3.0, the Unix cache manager has used the root identity credentials to create anonymous outgoing connections to the location service and each fileserver. However, if uid 0 is assigned a token, then those Rx connections will no longer be anonymous. Beginning with this release anonymous outgoing connections are always created with the NOPAG identity (uid 0xffffffff) instead of the root identity.
- When establishing an outgoing rxgk connection, do not fallback to the systemuser's credentials if the user's credentials resulted in a fatal error. Falling back to the systemuser's credentials can result in inappropriate use of an anonymous connection.
- Improved access rights cache correctness for YFS servers
  In prior releases, the access check logic used the file rights for any files fetched from an AuriStorFS fileserver. For files fetched from an AFS-3 fileserver (and, historically, for all files), it used the directory rights, with the (a)dmin right from the file mixed in. The (a)dmin right on a non-directory indicates that the object is owned by the authenticated user.
  This approach has some issues when combined with the access rights cache, and current fileserver callback behavior. On an AuriStorFS file server, the rights on a non-directory may be determined by the rights granted on its parent directory or, with per-file ACLs, those granted on the object itself. The fileserver will only break a non-directory's callback when a per-file ACL is changed - changing a directory ACL will not break callbacks on files within that directory. This means that changing a directory ACL will not invalidate access rights cache entries on files in tha directory, even if the effective ACL on this files has changed, and the cached rights are no longer correct.
  This release works around this by adding a new function which returns the access rights for a file hosted on an AuriStor fileserver. It uses the parent vnode information to locate the parent directory. If the parent directory isn't in the cache, or it doesn't have a valid callback, or if it has been changed since the file's access rights were cached, it clears the current access rights. Files without a parent directory must have per-file ACLs, and so their cached rights can be safely used.
  Note that files with parent vnodes may still have per-file ACLs, and that the breadcrumbing performed by the client may add parent vnode fields to vnodes which don't have them provided by the fileserver. Such vnodes may have their cached access rights cleared more frequently than necessary.
- Add a new mechanism for caching access rights within the vcache structure. This cache is protected via a vcache specific spinlock, and can be accessed without holding the GLOCK.
  This new cache mechanism returns the memory associated with cached rights back to the kernel's slab free memory pool instead of adding the unused rights structures to a cache manager managed free list. The previous cache implementation never returned allocated memory to the kernel. Instead, invalidated access rights were appended to a free access rights queue for later reuse.
- When a volume is accessed via multiple mountpoints a choice must be made regarding which mountpoint is considered to be the active (or parent) mountpoint. This release alters the behavior such that the active mountpoint is set every time a mountpoint is traversed.
  This behavior is easier to understand and is more likely to provide the expected result for a single process that repeatedly accesses volumes from multiple mountpoints. However, it can result in unexpected results when multiple processes are traversing multiple mountpoints in parallel without any synchronization.

v2021.05-45 (Not released)

v2021.05-44a (18 September 2024)

Authentication:
- AuriStorFS v2021.05-44 included an updated version of the Heimdal Kerberos framework used by AuriStorFS when acquiring yfs-rxgk and rxkad authentication tokens. The updated Heimdal included a bug which disabled the use of DNS SRV records for KDC discovery and DNS TXT records for realm discovery. As a side effect token acquisition might fail with an unable to reach any KDC in realm error. This is fixed in v2021.05-44a.

v2021.05-44 (17 August 2024)

Cache Manager:
- Since v0.192 the cache manager has failed acquire the global lock when upgrading a shared-lock to a write-lock during the execution of a background cache chunk file truncation.
Authentication:
- Neither MIT nor Heimdal gssapi nor their gss mechanisms consistently initialize the output 'minorStatus' parameter. Various functions can return either success or failure majorStatus values with minorStatus unassigned. As a result, stack garbage will be used when generating error messages. From now on libyfs_acquire will always initialize the minorStatus output variable to zero before calling into the gssapi library.
Command Parser:
- No longer accept the token "-" as a switch which eventually fails with a CMD_UNKNOWNSWITCH error. Instead, process the token as a data value.
- Optimize the processing of the loop which processes "source" command input.
- If the source command input file is "-", read from stdin.

v2021.05-41 (26 June 2024)

Rx Networking (libyfs_rx):
- A race during event creation can lead to the freeing of the event while its still in use.
- RFC1122 says that Net and Host unreachable ICMP errors might be transient and should therefore not be treated as fatal. There is no such language for the equivalent ICMPV6 errors. However, in practice ICMP6_DST_UNREACH_NOROUTE, ICMP6_DST_UNREACH_BEYONDSCOPE, and ICMP6_DST_UNREACH_ADDR can be transient.
  Linux has considered these ICMPV6 destination unreachable errors as non-fatal going back at least as far as the initial git repository commit.
  AuriStor Rx has always treated these as fatal errors which results in immediate termination of in-flight calls when received. Even if the network route corrects itself before the call timeout period expires. This release mirrors the Linux behavior and makes these errors non-fatal.
Cache Manager:
- For the first time the cache manager can detect the deletion of a volume and handle the creation of a new volume with the same name but a different volume id.
- If the location service reports the deletion of a volume, invalidate all mount points to that volume.
- RXAFS_GetCapabilities RPC failures should not be treated as a fatal error preventing failover to another replica site.
Authentication ("libyfs_acquire") used by aklog, vos, pts, bos, afsio:
- rxkad_k5 token acquisition krb5 ccache management. This release altered the krb5 credential cache management strategy once again to work around different bugs in MIT krb5 and Heimdal.
- New ACQUIRE_ERR_CRED_EXPIRED error code introduced to represent the case when a request for a service credential returns one that is already expired.
Command parser (libyfs_cmd):
- When parsing configuration files there is a depth limit of ten active inclusions. This limit was improperly enforced as a limit of ten included files instead of a depth of ten included files. As of this release it is now possible to populate an includedir directory with any number of .conf files.

v2021.05-40

Not released.

v2021.05-39 (20 May 2024)

Parallel Random Number Generation:
- AuriStorFS processes rely upon the krb5_generate_random() and RAND_bytes() functions to obtain random bytes for cryptographic operations and random counters. krb5_generate_random() internally acquires a mutex to protect internal state information. This mutex has become a significant barrier to the encryption and checksumming of Rx packets with both yfs-rxgk and rxkad.
  
  This release replaces general use of krb5_generate_random() and RAND_bytes() with a per-thread ChaCha20 CS-PRNG. This avoids the acquisition of a global mutex and permits increased parallelism on multi-core systems.
Rx Networking (libyfs_rx):
- The Rx network stack schedules a garbage collection operation to execute once per minute. This operation enforces call timeouts, destroys idle connections and destroys idle peers. The operation has historically been performed by the Rx event thread which is already responsible for performing actions in response to call RTOs, sending NAT Ping and keep-alive packets, and retrying connection challenge and reachability checks.
  
  The time complexity of the garbage collection operation is determined by the number of calls, connections, and peers. The busier the Rx endpoint the more work must be performed during each garbage collection run and the longer it takes to complete. While garbage collection is active other events cannot be processed which can interfere with the proper flow control of active calls.
  
  As with all Rx events, the garbage collection event is scheduled to execute at an absolute clock time. If the system clock drifts (or is administratively set) backwards garbage collection will not be performed until the clock catches up with the scheduled time.
  
  Another responsibility of the garbage collection procedure is to terminate calls if the system clock drifted backwards by five minutes or longer. However, when the clocked drifts backwards garbage collection is not performed until the clock has advanced beyond the point where calls require termination. As a result, calls are not terminated due to backwards clock drift and they can stall.
  
  This release re-implements the garbage collection procedure using a dedicated thread and relative waits. This change ensures that the garbage collection procedure will not prevent the execution of call related events and permits calls to be terminated when large backward clock drifts are detected.
Disk Cache Management:
- Since IBM AFS 3.5, the cache has been considered "too full" even if there exist cache files have been discarded but not yet truncated. When the cache is "too full" most operations that write to the cache will block until truncation of discarded cache files has been performed which results in unnecessary delays. This release fixes the cache such that that discarded but not yet truncated cache files do not block write operations.
- This release permits the cache truncation daemon thread to exit sooner if the cache manager is shutting down.
Improved failover when the RXGK service (co-located with each vlserver) fails to issue tokens. The failures might be the result of misconfiguration, an inability to read keys or loss of Ubik quorum.

v2021.05-38 (29 February 2024)

As with other AuriStorFS releases since the beginning of 2024, this release includes additional improvements to the Rx RPC implementation which are related to the possibility of silent data corruption when Rx jumbograms are in use. Prior releases disabled the negotiation of Rx jumbograms such that the v2021.05-37 Rx peer will refuse to send Rx jumbograms and will request that the remote peer does not send them. However, a bad actor could choose to send Rx jumbograms even though they were asked not to. v2021.05-38 introduces additional protections to ensure that a corrupt Rx jumbogram is dropped instead of being accepted.

The v2021.05-38 Rx RPC implementation also includes two optimizations. First, when Rx initiators complete a call they will no longer send an extra ACK packet to the Rx acceptor of the completed call. The sending of this unnecessary ACK creates additional work for the server which can result in increased latency for other calls being processed by the server.

Second, all AuriStor Rx services require a reach check for incoming calls from Rx peers to help protect against Distributed Reflection Denial of Service (DRDoS) attacks and execution of RPCs when the response cannot be delivered to the caller. A new reach check is required for each new call that arrives more than 60 seconds after the prior reach check completed. v2021.05-38 Rx considers the successful acknowledgment of a response DATA packet as a reach check validation. With this change reach checks will not be periodically required for a peer that completes at least one call per 60 seconds. A 1 RTT delay is therefore avoided each time a reach check can be avoided. In addition, reach checks require the service to process an additional ACK packet. Eliminating a large number of reach checks can improve overall service performance.

The final Rx RPC change in this release is specific to kernel implementations. Prior releases restricted the frequency of executing time scheduled Rx events to a granularity no smaller than 500ms. As a result an RTO timer event for a lost packet could not be shorter than 500ms even if the measured RTT for the connection is significantly smaller. The minimum RTO for a connection in AuriStor Rx is 200ms. The inability to schedule shorter timeouts impacts recovery from packet loss.

v2021.05-37 (5 February 2024)

Rx improvements:
- The v2021.05-36 release permanently disabled all use of Rx jumbograms due to a risk of silent data corruption. However, when advertising the number of acceptable datagrams in the ACK trailer a missing htonl() set the value to 16777216 instead of 1 on little-endian systems.
- When sending a PING ACK as a reachability test, ensure that the previousPacket field is properly assigned to the largest accepted DATA packet sequence number instead of zero.
- Replace the initialization state flag with two flags. One that indicates that Rx initialization began and the other that it succeeded. The first prevents multiple attempts at initialization after failure. The second prevents shutdown from accessing uninitialized structures if initialization failed.
Cache Manager Improvements:
- No longer refuse to start if both the 'cachedir' and 'memcache' options are present in the configuration file.
- New variable to store the maximum number of cache blocks used. which is accessible via /proc/fs/auristorfs/cache/blocks_used_max.

v2021.05-36 (10 January 2024)

Rx improvements:
- Permanently disable all use of Rx jumbograms due to a risk of silent data corruption.
  
  Ever since OpenAFS 1.0, and possibly before, a race condition has existed when Rx transmits packets. As the rx_call.lock is dropped when starting packet transmission, there is no protection for data that is being copied into the kernel by sendmsg(). It is critical that this packet data is not modified by another thread. However, races exist between the application, listener, and event threads that can lead to retransmissions starting whilst an original transmission is still in progress. This can lead to the packet headers being overwritten, and either the original transmission, the retransmission or both sending corrupt data to the peer.
  
  This corruption can affect the packet serial number or packet flags. It is particularly harmful when the packet flags are corrupted, as this can lead to multiple Rx packets which were intended to be sent as Rx jumbograms being delivered and misinterpreted as a single large packet. The eventual result of this depends on the Rx security class in play, but it can cause decrypt integrity errors (rxgk:crypt and rxgk:auth) or corruption of the data stream (rxnull, rxgk:clear or rxkad:auth).
  
  All AuriStorFS servers, OpenAFS 1.6 or later servers, and the Windows cache manager have been shipped with Rx jumbograms disabled by default. The UNIX cache managers however are shipped with jumbograms enabled. There are many AFS cells around the world that continue to deploy OpenAFS 1.4 or earlier fileservers which continue to negotiate the use of Rx jumbograms.
  
  It is worth noting that all AuriStorFS v0.198 and later fileservers and cache managers implement explicit checks that will recognize the corrupted application data stream and prevent corrupted file content from being stored either into an AFS vnode's backing store in the volume's object store or the cache manager's AFS cache. OpenAFS cache managers and fileservers do not have these protections.
  
  With Rx jumbograms disabled the maximum number of Rx packets in a datagram is reduced from 6 to 1; the maximum number of send and receive datagram fragments is reduced from 4 to 1; and the maximum advertised MTU is restricted to 1444 - the maximum rx packet size prior to the introduction of jumbograms in IBM AFS 3.5.
- If the rx call flow state transitions from either the RECOVERY or RESCUE states to the LOSS state as a result of an RTO resend event while writing packets to the network, cease transmission of any new DATA packets if there are packets in the resend queue.
- When the call flow state is LOSS and all packets in the resend queue have been retransmitted and yet the recovery point has not been reached, then permit new DATA packets to be sent in order to maintain a full congestion window.
- Add a safety check to prevent the estimated RTT from underflowing when the actual roundtrip time is smaller than 125us.
- Fix the computation of the padding required for rxgk encrypted packets. This bug results in packets sending 8 bytes fewer per packets than the network permits. This bug accidentally prevented the construction of Rx jumbograms when a call is protected by rxgk:crypt.
- Replace the random number generator with a more security source of random bytes.

v2021.05-33 (27 November 2023)

Rx improvements:
- Not all calls transfer enough data to be able to measure a smoothed round-trip time (SRTT). Calls which are unable to compute a SRTT should not be used to update the peer host RTO value which is used to initialize the RTO for subsequent calls.
  
  Without this change, a single DATA packet call will cause the peer host RTO to be reduced to 0ms. Subsequent calls will start with a RTO value of MAX(0, rxi_minPeerTimeout) where rxi_minPeerTimeout defaults to 200ms. If the actual measured RTO is greater than 200ms, then initial RTO will be too small resulting in premature triggering of the RTO timer and the call flow state entering the loss phase which can significantly hurt performance.
- Initialize the peer host RTO to rxi_minPeerTimeout (which defaults to 200ms) instead of one second. Although RFC6298 recommends the use of one second when no SRTT is available, Rx has long used the rxi_minPeerTimeout value for other purposes which are supposed to be consistent with initial RTO value. It should be noted that Linux TCP uses 200ms instead of one second for this purpose.
- If associating a security class with an Rx connection fails immediately place the Rx connection into an error state. A failure might occur if the security class is unable to access valid key material.
- If an incoming Rx call requires authentication and the security class is unable to successfully generate a challenge, put the incoming Rx connection into an error state and issue an abort to the caller.
- If an incoming Rx call requires authentication and the security class is able to generate a challenge but the challenge cannot be returned to Rx, then treat this as a transient error. Do not acknowledge the incoming DATA packet and do not place the Rx connection into an error state. An attempt to re-issue the challenge will be performed when the DATA packet is retransmitted.
- If an Rx call is terminated due to the expiration of the configured connection dead time, idle dead time, hard dead time, or as a result of clock drift, then send an ABORT to the peer notifying them that the call has been terminated. This is particularly important for terminated outgoing calls. If the peer does not know to terminate the call, then the call channel might be in use when the next outgoing call is issued using the same call channel. If the next incoming call is received by an in-use call channel, the receiver must drop the received DATA packet and return a BUSY packet. The call initiator will need to wait for a retransmission timeout to pass before retransmitting the DATA packet. Receipt of BUSY packets cannot be used to keep a call alive and therefore the requested call is at greater risk of timing out if the network path is congested.
aklog and krb5.log (via libyfs_acquire):
- If the linked Kerberos library implements krb5_cc_cache_match() and libacquire has been told to use an explicit principal name and credential cache, the Kerberos library might return KRB5_CC_NOTFOUND even though the requested credential cache is the correct one to use. This release will not call krb5_cc_cache_match() if the requested credential cache contains the requested principal.
Cell Service Database (cellservdb.conf):
- cellservdb.conf has been synchronized with the 31 Oct 2023 update to the grand.central.org CellServDB file.

v2021.05-32 (9 October 2023)

No significant changes for macOS compared to v2021.05-31

v2021.05-31 (25 September 2023)

New platform:
- macOS 14 Sonoma
macOS 14 Sonoma:

AuriStorFS v2021.05-29 and later installers for macOS 13 Ventura are compatible with macOS 14 Sonoma and do not need to be removed before upgrading to macOS 14 Sonoma. Installation of the macOS 14 Sonoma version of AuriStorFS is recommended.

Cache Manager:
- If an AuriStorFS cache manager is unable to use the yfs-rxgk security class when communicating with an AuriStorFS fileserver, it must assume it is IBM AFS 3.6 or OpenAFS and upgrade it to AuriStorFS if an upgrade probe returns a positive result. Once a fileserver's type is identified as AuriStorFS the type should never be reset; even if communication with the fileserver is lost or the fileserver restarts.
  
  If an AuriStorFS fileserver is replaced by an OpenAFS fileserver on the same endpoint, then the UUID of the OpenAFS must be different. As a result, the OpenAFS fileserver will be observed as distinct from the AuriStorFS fileserver that previously shared the endpoint.
  
  Prior to this release there were circumstances in which the cache manager discarded the fileserver type information and would fail to recognize the fileserver as an AuriStorFS fileserver when yfs-rxgk could not be used. This release prevents the cache manager from resetting the type information if the fileserver is marked down.
- If a fileserver's location service entry is updated with a new uniquifier value (aka version number), this indicates that one of the following might have changed:
  1. the fileserver's capabilities
  2. the fileserver's security policy
  3. the fileserver's knowledge of the cell-wide yfs-rxgk key
  4. the fileserver's endpoints
  Beginning with this release the cache manager will force the establishment of new Rx connections to the fileserver when the uniquifier changes. This ensures that the cache manager will attempt to fetch new per-fileserver yfs-rxgk tokens from the cell's RXGK service, enforce the latest security policy, and not end up in a situation where its existing tokens cannot be used to communicate with the fileserver.
aklog:
- Fix incorrect output when populating the server list for a service fails. The stashed extended error explaining the cause of the failure was not displayed.
- If a cell has neither _afs3-prserver._udp. DNS SRV records nor AFSDB records, the lookup of the cell's protection servers would fail if there is no local cell configuration details. The fallback to use _afs3-vlserver._udp. DNS SRV records did not work. This is corrected in this release.

v2021.05-30 (6 September 2023)

Do not mark a fileserver down in response to a KRB5 error code.
fs cleanacl must not store back to the file server a cleaned acl if it was inherited from a directory. Doing so will create a file acl.
Correct the generation of never expire rxkad_krb5 tokens from Kerberos v5 tickets which must have a start time of Unix epoch and an end time of 0xFFFFFFFF seconds. The incorrectly generated tokens were subject to the maximum lifetime of 30 days.
Correct the generation of the yfs-rxgk RESPONSE packet header which failed to specify the key version generation number used to encrypt the authenticator. If the actual key version is greater than zero, then the authenticator would fail to verify.
Enforce a maximum NAT ping period of 20s to ensure that NAT/PAT/firewall rules due not expire while Rx RPCs are in-flight.

v2021.05-29 (26 June 2023)

Execution of fs commands such as examine, whereis, listquota, fetchacl, cleanacl, storeacl, whoami, lsmount, bypassthreshold and getserverprefs could result in memory leaks by the AuriStorFS kernel extension.

v2021.05-27 (1 May 2023)

Fixes for bugs in vos introduced in v2021.05-26.

v2021.05-26 (17 April 2023)

Fixed a potential kernel memory leak when triggered by fs examine, fs listquota, or fs quota.
Increased logging of VBUSY, VOFFLINE, VSALVAGE, and RX_RESTARTING error responses. A log message is now generated whenever a task begins to wait as a result of one of these error responses from a fileserver. Previously, a message was only logged if the volume location information was expired or discarded.
Several changes to optimize internal volume lookups.
Faster failover to replica sites when a fileserver returns RX_RESTARTING, VNOVOL or VMOVED.
rxdebug regains the ability to report rx call flags and rx_connection flags.
The RXRPC library now terminates calls in the QUEUED state when an ABORT packet is received. This clears the call channel making it available to accept another call and reduces the work load on the worker thread pool.
Fileserver endpoint registration changes no longer result in local invalidation of callbacks from that server.
Receipt of an RXAFSCB_InitCallBackState3 RPC from a fileserver no longer resets the volume site status information for all volumes on all servers.

v2021.05-25 (28 December 2022)

The v2021.05-25 release includes further changes to RXRPC to improve reliability. The changes in this release prevent improper packet size growth. Packet size growth should never occur when a call is attempting to recover from packet loss; and is unsafe when the network path's maximum transmission unit is unknown. Packet size growth with be re-enabled in a future AuriStorFS release that includes Path MTU detection and the Extended SACK functionality.
Improved error text describing the source of invalid values in /etc/yfs/yfs-client.conf or included files and directories.

v2021.05-24 (25 October 2022)

New Platform: macOS 13 (Ventura)

v2021.05-22 (12 September 2022) and v2021.05-21 (6 September 2022)

RX RPC
- If receipt of a DATA packet causes an RX call to enter an error state, do not send the ACK of the DATA packet following the ABORT packet. Only send the ABORT packet.
- AuriStor RX has failed to count and report the number of RX BUSY packets that have been sent. Beginning with this change the sent RX BUSY packet count is once again included in the statistics retrieved via rxdebug server port -rxstats.
- Introduce minimum and maximum bounds checks on the ACK packet trailer fields. If the advertised values are out of bounds for the receiving RX stack, do not abort the call but adjust the values to be consistent with the local RX RPC implementation limits. These changes are necessary to handle broken RX RPC implementations or prevent manipulation by attackers.

New to v2021.05-20 (15 August 2022) and v2021.05-19 (13 August 2022)

RX RPC
- Include the DATA packet serial number in the transmitted reachability check PING ACK. This permits the reachability test ACK to be used for RTT measurement.
- Do not terminate a call due to an idle dead timeout if there is data pending in the receive queue when the timeout period expires. Instead deliver the received data to the application. This change prevents idle dead timeouts on slow lossy network paths.
- Fix assignment of RX DATA, CHALLENGE, and RESPONSE packet serial numbers in macOS (KERNEL). Due to a mistake in the implementation of atomic_add_and_read the wrong serial numbers were assigned to outgoing packets.

New to v2021.05-18 (12 July 2022)

Cache Manager
- Prevent a kernel memory leak of less than 64 bytes for each bulkstat RPC issued to a fileserver. Bulkstat RPCs can be frequently issued and over time this small leak can consume a large amount of kernel memory. Leak introduced in AuriStorFS v0.196.
- The Perl::AFS module directly executes pioctls via the OpenAFS compatibility pioctl interface instead of the AuriStorFS pioctl interface. When Perl::AFS is used to store an access control list (ACL), the deprecated RXAFS_StoreACL RPC would be used in place of the newer RXAFS_StoreACL2 or RXYFS_StoreOpaqueACL2 RPCs. This release alters the behavior of the cache manager to use the newer RPCs if available on the fileserver and fallback to the deprecated RPC. The use of the deprecated RPC was restricted to use of the OpenAFS pioctl interface.
RX RPC
- Handle a race during RX connection pool probes that could have resulted in the wrong RX Service ID being returned for a contacted service. Failure to identify that correct service id can result in a degradation of service.
- The Path MTU detection logic sends padded PING ACK packets and requests a PING_RESPONSE ACK be sent if received. This permits the sender of the PING to probe the maximum transmission unit of the path. Under some circumstances attempts were made to send negative padding which resulted in a failure when sending the PING ACK. As a result, the Path MTU could not be measured. This release prevents the use of negative padding.
Preparation for supporting macOS 13 Ventura when it is released in Fall 2022.
Some shells append a slash to an expanded directory name in response to tab completion. These trailing slashes interfered with "fs lsmount", "fs flushmount" and "fs removeacl" processing. This release includes a change to prevent these commands from breaking when presented a trailing slash.

New to v2021.05-17 (16 May 2022)

Cell Service Database Updates
- Update cern.ch, ics.muni.cz, ifh.de, cs.cmu.edu, qatar.cmu.edu, it.kth.se
- Remove uni-hohenheim.de, rz-uni-jena.de, mathematik.uni-stuttgart.de, stud.mathematik.uni-stuttgart.de, wam.umd.edu
- Add ee.cooper.edu
- Restore ams.cern.ch, md.kth.se, italia
Fix parsing of [afsd] rxwindow configuration which can be used to specified a non-default send/receive RX window size. The current default is 128 packets.
RX Updates
- Add nPacketsReflected

Logging improvements

Cache directory validation errors log messages now include the cache directory path.

Log the active configuration path if "debug" logging is enabled.

More details of rxgk token extraction failures.

New to v2021.05-16 (24 March 2022)

RX - Previous releases re-armed the Retransmission Timeout (RTO) each time a new unacknowledged packet is acknowledged instead of when a new leading edge packet is acknowledged. If leading edge data packet and its retransmission are lost, the call can remain in the "recovery" state where it continues to send new data packets until one of the following is true:
. the maximum window size is reached
. the number of lost and resent packets equals 'cwind'
at which point there is nothing left to transmit. The leading edge data packet can only be retransmitted when entering the "loss" state but since the RTO is reset with each acknowledged packet the call stalls for one RTO period after the last transmitted data packet is acknowledged.

This poor behavior is less noticiable with small window sizes and short lived calls. However, as window sizes and round-trip times increase the impact of a twice lost packet becomes significant.

RX - Never set the high-order bit of the Connection Epoch field. RX peers starting with IBM AFS 3.1b through AuriStor RX v0.191 ignore the source endpoint when matching incoming packets to RX connections if the high-order epoch bit is set. Ignoring the source endpoint is problematic because it can result in a call entering a zombie state whereby all PING ACK packets are immediately responded to the source endpoint of the PING ACK but any delayed ACK or DATA packets are sent to the endpoint bound to the RX connection. An RX client that moves from one network to another or which has a NAT|PAT device between it and the service can find themselves stuck.

Starting with AuriStor RX v0.192 the high-order bit is ignored by AuriStor RX peer when receiving packets. This change to always clear the bit prevents IBM AFS and OpenAFS peers from ignoring the source endpoint.

RX - The initial packetSize calculation for a call is altered to require that all constructed packets before the receipt of the first ACK packet are eligible for use in jumbograms if and only if the local RX stack has jumbograms enabled and the maximum MTU is large enough. By default jumbograms are disabled for all AuriStorFS services. This change will have a beneficial impact if jumbograms are enabled via configuration; or when testing RX performance with "rxperf".

New fs whereis -noresolve option displays the fileservers by network endpoint instead of DNS PTR record hostname.

New to v2021.05-15 (24 January 2022)

kernel - fixed YFS_RXGK service rx connection pool leak

New to v2021.05-14 (20 January 2022)

fs mkmount permit mount point target strings longer than 63 characters.
afsd enhance logging of yfs-rxgk token renewal errors.

afsd gains a "principal = configuration option for use with keytab acquisition of yfs-rxgk tokens for the cache manager identity.

kernel - Avoid unnecessary rx connection replacement by racing threads after token replacement or expiration.

kernel - Fix a regression introduced in v2021.05 where an anonymous combined identity yfs-rxgk token would be replaced after three minutes resulting in the connection switching from yfs-rxgk to rxnull.

kernel - Fix a regression introduced in v0.208 which prevented the invalidation of cached access rights in response to a fileserver callback rpc. The cache would be updated after the first FetchStatus rpc after invalidation.

kernel - Reset combined identity yfs-rxgk tokens when the system token is replaced.

kernel - The replacement of rx connection bundles in the cache manager to permit more than four simultaneous rx calls per uid/pag with trunked rx connections introduced the following regressions in v2021.05.

a memory leak of discarded rx connection objects

failure of NAT ping probes after replacement of an connection

inappropriate use of rx connections after a service upgrade failure

All of these regressions are fixed in patch 14.

New to v2021.05-12 (7 October 2021)

fs ignorelist -type afsmountdir in prior releases could prevent access to /afs.

Location server rpc timeout restored to two minutes instead of twenty minutes.

Location server reachability probe timeout restored to six seconds instead of fifty seconds.

Cell location server upcall results are now cached for fifteen seconds.

Multiple kernel threads waiting for updated cell location server reachability probes now share the results of a single probe.

RX RPC implementation lock hierarchy modified to prevent a lock inversion.

RX RPC client connection reference count leak fixed.

RX RPC deadlock during failed connection service upgrade attempt fixed..

New to v2021.05-9 (25 October 2021)

First public release for macOS 12 Monterey build using XCode 13. When upgrading macOS to Monterey from earlier macOS releases, please upgrade AuriStorFS to v2021.05-9 on the starting macOS release, upgrade to Monterey and then install the Monterey specific v2021.05-9 release.

Improved logging of "afsd" shutdown when "debug" mode is enabled.

Minor RX network stack improvements

New to v2021.05-3 (10 June 2021)

Fix for [cells] cellname = {...} without server list.

New to v2021.05 (31 May 2021)

Multi-homed location servers are finally managed as a single server instead of treating each endpoint as a separate server. The new functionality is a part of the wholesale replacement of the former cell management infrastructure. Location server communication is now entirely managed as a cluster of multi-homed servers for each cell. The new infrastructure does not rely upon the global lock for thread safety.

This release introduces a new infrastructure for managing user/pag entities and tracking their per cell tokens and related connection pools.

Expired tokens are no longer immediately deleted so that its possible for them to be listed by "tokens" for up to two hours.

Prevent a lock inversion introduced in v0.208 that can result in a deadlock involving the GLOCK and the rx call.lock. The deadlock can occur if a cell's list of location servers expires and during the rebuild an rx abort is issued.

Add support for rxkad "auth" mode rx connections in addition to "clear" and "crypt". "auth" mode provides integrity protection without privacy.

Add support for yfs-rxgk "clear" and "auth" rx connection modes.

Do not leak a directory buffer page reference when populating a directory page fails.

Re-initialize state when populating a disk cache entry using the fast path fails and a retry is performed using the slow path. If the data version changes between the attempts it is possible for truncated disk cache data to be treated as valid.

Log warnings if a directory lookup operation fails with an EIO error. An EIO error indicates that an invalid directory header, page header, or directory entry was found.

Do not overwrite RX errors with local errors during Direct-I/O and StoreMini operations. Doing so can result in loss of VBUSY, VOFFLINE, UAENOSPC, and similar errors.

Correct a direct i/o code path which could overwrite a fileserver returned error code with a local RXGEN_CC_UNMARSHAL error.

Correct the StoreMini code path which could overwrite a fileserver returned error code with a local RXGEN_CC_UNMARSHAL error.

Ensure the rx call object is not locked when writing to the network socket.

Removed all knowledge of the KERNEL global lock from RX. Acquiring the GLOCK from RX is never safe if any other lock is held. Doing so is a lock order violation that can result in deadlocks.

Fixed a race in the opr_reservation system that could produce a cache entry reference undercount.

If a directory hash chain contains a circular link, a buffer page reference could be leaked for each traversal.

Each AFS3 directory header and page header contains a magic tag value that can be used in a consistency check but was not previously checked before use of each header. If the header memory is zero filled during a lookup, the search would fail producing an ENOENT error. Starting with this release the magic tag values are validated on each use. An EIO error is returned if there is a tag mismatch.

"fs setcrypt -crypt auth" is now a permitted value. The "auth" mode provides integrity protection but no privacy protection.

Add new "aklog -levels " option which permits requesting "clear" and "auth" modes for use with yfs-rxgk.

Update MKShim to Apple OpenSource MITKerberosShim-79.

Report KLL errors via a notification instead of throwing an exception which (if not caught) will result in process termination.

If an exception occurs while executing "unlog" catch it and ignore it. Otherwise, the process will terminate.

New to v2021.04 (22 April 2021)

Primarily bug fixes for issues that have been present for years.

A possibility of an infinite kernel loop if a rare file write / truncate pattern occurs.

A bug in silly rename handling that can prevent cache manager initiated garbage collection of vnodes.

New to v0.209 (13 March 2021)

fs setserverprefs and fs getserverprefs updated to support IPv6 and CIDR specifications.

Improved error handling during fetch data and store data operations.

Prevents a race between two vfs operations on the same directory which can result in caching of out of date directory contents.

Use cached mount point target information instead of evaluating the mount point's target upon each access.

Avoid rare data cache thrashing condition.

Prevent infinite loop if a disk cache error occurs after the first page in a chunk is written.

Network errors are supposed to be returned to userspace as ETIMEDOUT. Previously some were returned as EIO.

When authentication tokens expire, reissue the fileserver request anonymously. If the anonymous user does not have permission either EACCES or EPERM will be returned as the error to userspace. Previously the vfs request would fail with an RXKADEXPIRED or RXGKEXPIRED error.

If growth of an existing connection vector fails, wait on a call slot in a previously created connection instead of failing the vfs request.

Volume and fileserver location query infrastructure has been replaced with a new modern implementation.

Replace the cache manager's token management infrastructure with a new modern implementation.

New to v0.206 (12 January 2021) - Bug fixes

Prevents a possible panic during unmount of /afs.

Improved failover and retry logic for offline volumes.

New to v0.205 (24 December 2020) - Bug fixes

Volume name-to-id cache improvements

Fix expiration of name-to-id cache entries

Control volume name-to-id via sysctl

Query volume name-to-id statistics via sysctl

Improve error handling for offline volumes

Fix installer to prevent unnecessary installation of Rosetta 2 on Apple Silicon

New to v0.204 (25 November 2020) - Bug fix for macOS Big Sur

v0.204 prevents a kernel panic on Big Sur when AuriStorFS is stopped and restarted without an operating system reboot.

introduces a volume name-to-id cache independent of the volume location cache.

New to v0.203 (13 November 2020) - Bug fix for macOS

v0.203 prevents a potential kernel panic due to network error.

New to v0.201 (12 November 2020) - Universal Big Sur (11.0) release for Apple Silicon and Intel

v0.201 introduces a new cache manager architecture on all macOS versions except for High Sierra (10.12). The new architecture includes a redesign of:

kernel extension load

kernel extension unload (not available on Big Sur)

/afs mount

/afs unmount

userspace networking

The conversion to userspace networking will have two user visible impacts for end users:
The Apple Firewall as configured by System Preferences -> Security & Privacy -> Firewall is now enforced. The "Automatically allow downloaded signed software to receive incoming connections" includes AuriStorFS.

Observed network throughput is likely to vary compared to previous releases.

On Catalina the "Legacy Kernel Extension" warnings that were displayed after boot with previous releases of AuriStorFS are no longer presented with v0.201.

AuriStorFS /afs access is expected to continue to function when upgrading from Mojave or Catalina to Big Sur. However, as AuriStorFS is built specifically for each macOS release, it is recommended that end users install a Big Sur specific AuriStorFS package.
AuriStorFS on Apple Silicon supports hardware accelerated aes256-cts-hmac-sha1-96 and aes128-cts-hmac-sha1-96 using AuriStor's proprietary implementation.

New to v0.200 (4 November 2020) - Final release for macOS El Capitan (10.11)

The network path between a client and a server often traverses one or more network segments separated by NAT/PAT devices. If a NAT/PAT times out a RPCs endpoint translation mid-call, this can result in an extended delay before failure and the server being marked down, or worse, a call that never terminates and a client that appears to hang until the fileserver is restarted.

This release includes significant changes to the RX stack and the UNIX cache manager to detect such conditions, fail the calls quickly and detect when it is safe to retry the RPC.

NAT/PAT devices that drop endpoint mappings while in use are anti-social and can result in unwanted delays and even data loss. Whenever possible they should be avoided when they can be. That said, the changes in this release are a huge step to making the loss of endpoint mappings tolerable.

Fix segmentation fault of Backgrounder when krb5_get_credentials() fails due to lack of network connectivity.

Fix the "afsd" rxbind option which was ignored if the default port, 7001, is in use by another process on the system.

If a direct i/o StoreData or FetchData RPC failed such that it must be retried, the retried RPC would fail due to an attempt to Fetch or Store the wrong amount of data. This is fixed.

Servers are no longer marked down if RPCs fail with RX_CALL_PEER_RESET, RX_CALL_EXCEEDS_WINDOW, or RX_PROTOCOL_ERROR. RPCs that are safe to retry are retried.

Fixed a race between a call entering error state and call completion that can result in a the call remaining in the DALLY state and the connection channel remaining in use. If this occurs during process or system shutdown it can result in a deadlock.

During shutdown cancel any pending delayed aborts to prevent a potential deadlock. If a deadlock occurs when unloading a kernel module a reboot will be required.

Updated cellservdb.conf

New to v0.197.1 (31 August 2020) and v0.198 (10 October 2020)

Prevent Dead vnode has core/unlinkedel/flock panic introduced in v0.197.

New to v0.197 (26 August 2020)

A new callback management framework for UNIX cache managers reduces the expense of processing volume callback RPCs from O(number of vcache objects) to O(1). A significant amount of lock contention has been avoided. The new design reduces the risk of the single callback service worker thread blocking. Delays in processing callbacks on a client can adversely impact fileserver performance and other clients in the cell.

Bulk fetch status RPCs are available on macOS for the first time. Bulk fetch status permits optimistic caching of vnode status information without additional round-trips. Individual fetch status RPCs are no longer issued if a bulk status fails to obtain the required status information.

Hardware accelerated crypto is now available for macOS cache managers. AuriStor's proprietary aes256-cts-hmac-sha1-96 and aes256-cts-hmac-sha512-384 implementations leverage Intel processor extensions: AESNI AVX2 AVX SSE41 SSSE3 to achieve the fastest encrypt, decrypt, sign and verify times for RX packets.

This release optimizes the removal of "._" files that are used to store extended attributes by avoiding unnecessary status fetches when the directory entry is going to be removed.

When removing the final directory entry for an in-use vnode, the directory entry must be silly renamed on the fileserver to prevent removal of the backing vnode. The prior implementation risked blindly renaming over an existing silly rename directory entry.

Behavior change! When the vfs performs a lookup on ".", immediately return the current vnode.

if the object is a mount point, do not perform fakestat and attempt to resolve the target volume root vnode.

do not perform any additional access checks on the vnode. If the caller already knows the vnode the access checks were performed earlier. If the access rights have changed, they will be enforced when the vnode is used just as they would have if the lookup of "." was performed within the vfs.

do not perform a fetch status or fetch data rpcs. Again, the same as if the lookup of "." was performed within the vfs.

Volumes mounted at more than one location in the /afs namespace are problematic on more than one operating system that do not expect directories to have more than one parent. It is particularly problematic if a volume is mounted within itself. Starting with this release any attempt to traverse a mountpoint to the volume containing the mountpoint will fail with ENODEV.

When evaluating volume root vnodes, ensure that the vnode's parent is set to the parent directory of the traversed mountpoint and not the mountpoint. Vnodes without a parent can cause spurious ENOENT errors on Mojave and later.

v0.196 was not publicly released.

New to v0.195 (14 May 2020)

This is a CRITICAL update for AuriStorFS macOS clients.

In Sep 2019 AuriStorFS v0.189 was released which provided faster and less CPU intensive writing of (>64GB) large files to /afs. These improvements introduced a hash collision bug in the store data path of the UNIX cache manager which can result in file corruption. If a hash collision occurs between two or more files that are actively being written to via cached I/O (not direct I/O), dirty data can be discarded from the auristorfs cache before it is written to the fileserver creating a file with a range of zeros (a hole) on the fileserver. This hole might not be visible to the application that wrote the data because the lost data was cached by the operating system. This bug has been fixed in v0.195 and it is for this reason that v0.195 has been designated a CRITICAL release for UNIX/Linux clients.

While debugging a Linux SIGBUS issue, it was observed that receipt of an ICMP network error in response to a transmitted packet could result in termination of an unrelated rx call and could mark a server down. If the terminated call is a StoreData RPC, permanent data loss will occur. All Linux clients derived from the IBM AFS code base experience this bug. The v0.195 release prevents this behavior.

This release includes changes that impact all supported UNIX/Linux cache managers. On macOS there is reduced lock contention between kernel threads when the vcache limit has been reached.

The directory name lookup cache (DNLC) implementation was replaced. The new implementation avoids the use of vcache pointers which did not have associated reference counts, and eliminates the invalidation overhead during callback processing. The DNLC now supports arbitrary directory name lengths; the prior implementation only cached entries with names not exceeding 31 characters.

Prevent matching arbitrary cell name prefixes as aliases. For example "/afs/y" should not be an alias for "your-file-system.com". Some shells, for example "zsh", query the filesystem for names as users type. Delays between typed characters result in filesystem lookups. When this occurs in the /afs dynroot directory, this could result in cellname prefix string matches and the dynamic creation of directory entries for those prefixes.

New to v0.194 (2 April 2020)

This is a CRITICAL release for all macOS users. All prior macOS clients whether AuriStorFS or OpenAFS included a bug that could result in data corruption either when reading or writing.

This release also fixes these other issues:

sign and notarize installer plugin "afscell" bundle. The lack of digital signature prevented the installer from prompting for a cellname on some macOS versions.

prevent potential for corruption when caching locally modified directories.

v0.193 was withdrawn due to a newly introduced bug that could result in data corruption.

New to v0.192 (30 January 2020)

The changes improve stability, efficiency, and scalability. Post-0.189 changes exposed race conditions and reference count errors which can lead to a system panic or deadlock. In addition to addressing these deficiencies this release removes bottlenecks that restricted the number of simultaneous vfs operations that could be processed by the AuriStorFS cache manager. The changes in this release have been successfully tested with greater than 400 simultaneous requests sustained for for several days.

New to v0.191 (16 December 2019)

Restore keyed cache manager capability broken in v0.189.

Add kernel module version string to AuriStorFS Preference Pane.

Other kernel module bug fixes.

New to v0.190 (14 November 2019)

Short-circuit busy volume retries after volume or volume location entry is removed.

New to v0.189 (28 October 2019)

Faster "git status" operation on repositories stored in /afs.

Faster and less CPU intensive writing of (>64GB) large files to /afs. Prior to this release writing files larger than 1TB might not complete. With this release store data throughput is consistent regardless of file size. (See "UNIX Cache Manager large file performance improvements" later in this file).

macOS Catalina (8 October 2019)

AuriStorFS v0.188 released for macOS Catalina (10.15)

New to v0.188 (23 June 2019)

Increased clock resolution for timed waits from 1s to 1ns

Added error handling for rx multi rpcs interrupted by signals

New to v0.186 (29 May 2019)

v0.184 moved the /etc/yfs/cmstate.dat file to /var/yfs. With this change afsd would fail to start if /etc/yfs/cmstate.dat exists but contains invalid state information. This is fixed.

v0.184 introduced a potential deadlock during directory processing. This is fixed.

Handle common error table errors obtained outside an afs_Analyze loop. Map VL errors to ENODEV and RX, RXKAD, RXGK errors to ETIMEDOUT

Log all server down and server up events. Transition events from server probes failed to log messages.

RX RPC networking:

If the RPC initiator successfully completes a call without consuming all of the response data fail the call by sending an RX_PROTOCOL_ERROR ABORT to the acceptor and returning a new error, RX_CALL_PREMATURE_END, to the initiator.
Prior to this change failure to consume all of the response data would be silently ignored by the initiator and the acceptor might resend the unconsumed data until any idle timeout expired. The default idle timeout is 60 seconds.

Avoid transmitting ABORT, CHALLENGE, and RESPONSE packets with an uninitialized sequence number. The sequence number is ignored for these packets but set it to zero.

New to v0.184 (26 March 2019)

The initial congestion window has been reduced from 10 Rx packets to 4. Packet reordering and loss has been observed when sending 10 Rx packets via sendmmsg() in a single burst. The lack of udp packet pacing can also increase the likelihood of transmission stalls due to ack clock variation.

The UNIX Cache Manager underwent major revisions to improve the end user experience by revealing more error codes, improving directory cache efficiency, and overall resiliency. The cache manager implementation was redesigned to be more compatible with operating systems such as Linux and macOS that support restartable system calls. With these changes errors such as "Operation not permitted", "No space left on device", "Quota exceeded", and "Interrupted system call" can be reliably reported to applications. Previously such errors might have been converted to "I/O error".

New to v0.180 (9 November 2018)

RX reliability and performance improvements for high latency and/or lossy network paths such as public wide area networks.

A fix for a macOS firewall triggered kernel panic introduced in v0.177.

New to v0.177 (17 October 2018)

A fix to AuriStor's RX implementation bug introduced in v0.176 that interferes with communication with OpenAFS and IBM Location and File Services.

New to v0.176 (3 October 2018)

AuriStor's RX implementation has undergone a major upgrade of its flow control model. Prior implementations were based on TCP Reno Congestion Control as documented in RFC5681; and SACK behavior that was loosely modelled on RFC2018. The new RX state machine implements SACK based loss recovery as documented in RFC6675, with elements of New Reno from RFC5682 on top of TCP-style congestion control elements as documented in RFC5681. The new RX also implements RFC2861 style congestion window validation.

When sending data the RX peer implementing these changes will be more likely to sustain the maximum available throughput while at the same time improving fairness towards competing network data flows. The improved estimation of available pipe capacity permits an increase in the default maximum window size from 60 packets (84.6 KB) to 128 packets (180.5 KB). The larger window size increases the per call theoretical maximum throughput on a 1ms RTT link from 693 mbit/sec to 1478 mbit/sec and on a 30ms RTT link from 23.1 mbit/sec to 49.39 mbit/sec.

Improve shutdown performance by refusing to give up callbacks to known unreachable file servers and apply a shorter timeout period for the rest.

Permit RXAFSCB_WhoAreYou to be successfully executed after an IBM AFS or OpenAFS fileserver unintentionally requests an RX service upgrade from RXAFSCB to RXYFSCB.

RXAFS timestamps are conveyed in unsigned 32-bit integers with a valid range of 1 Jan 1970 (Unix Epoch) through 07 Feb 2106. UNIX kernel timestamps are stored in 32-bit signed integers with a valid range of 13 Dec 1901 through 19 Jan 2038. This discrepency causes RXAFS timestamps within the 2038-2106 range to display as pre-Epoch.

RX Connection lifecycle management was susceptible to a number of race conditions that could result in assertion failures, the lack of a NAT ping connection to each file server, and the potential reuse of RX connections that should have been discarded.

This release includes a redesigned lifecycle that is thread safe, avoids assertions, prevents NAT ping connection loss, and ensures that discarded connections are not reused.

The 0.174 release unintentionally altered the data structure returned to xstat_cm queries. This release restores the correct wire format.

Since v0.171, if a FetchData RPC fails with a VBUSY error and there is only one reachable fileserver hosting the volume, then the VFS request will immediately with an ETIMEDOUT error ("Connection timed out").

v0.176 corrects three bugs that contributed to this failure condition. One was introduced in v0.171, another in 0.162 and the final one dates to IBM AFS 3.5p1.

The intended behavior is that a cache manager, when all volume sites fail an RPC with a VBUSY error, will sleep for up to 15 seconds and then retry the RPC as if the VBUSY error had never been received. If the RPC continues to receive VBUSY errors from all sites after 100 cycles, the request will be failed with EWOULDBLOCK ("Operation would block") and not ETIMEDOUT.

Prefer VOLMISSING and VOLBUSY error states to network error states when generating error codes to return to the VFS layer. This will result in ENODEV ("No such device") errors when all volume sites return VNOVOL or VOFFLINE errors and EWOULDBLOCK ("Operation would block") errors when all volume sites return VBUSY errors. (v0.176)

New to v0.174 (24 September 2018)

macOS Mojave (10.14) support

New to v0.170 (27 April 2018)

Faster processing of cell configuration information by caching service name to port information.

RX call sequence number rollover to permit calls that require the transmission of more than 5.5TB of data.

Command parser Daylight Saving Time bug fix

Fix a bug that prevented immediate access to a mount point created with "fs mkmount" on the same machine.

Fix the setting of "[afsd] sysnames = " during cache manager startup.

New to v0.168 (6 March 2018)

Corrects "fs setacl -negative" processing [CVE-2018-7168]

Improved reliability for keyed cache managers. More persistent key acquisition renewals.

Major refresh to cellservdb.conf contents.

DNS SRV and DNS AFSDB records now take precedence when use_dns = yes

Kerberos realm hinting provided by
kerberos_realm = [REALM]

DNS host names are resolved instead of reliance on hard coded IP addresses
The cache manager now defaults to sparse dynamic root behavior. Only thiscell and those cells that are assigned aliases are included in /afs directory enumeration at startup. Other cells will be dynamically added upon first access.
Several other quality control improvements.

New to v0.167 (7 December 2017)

Addresses a critical remote denial of service vulnerability [CVE-2017-17432]
Alters the volume location information expiration policy to reduce the risk of single points of failures after volume release operations.
'fs setquota' when issued with quota values larger than 2TB will fail against OpenAFS and IBM AFS file servers
Memory management improvements for the memory caches.

New to v0.164 (11 November 2017)

Internal cache manager redesign. No new functionality.

New to v0.160 (21 September 2017)

Support for OSX High Sierra's new Apple File System (APFS). Customers must upgrade to v0.160 or later before upgrading to OSX High Sierra.
Reduced memory requirements for rx listener thread
Avoid triggering a system panic if an AFS local disk cache file is deleted or becomes inaccessible.
Fixes to "fs" command line output

New to v0.159 (7 August 2017)

Improved failover behavior during volume maintenance operations
Corrected a race that could lead the rx listener thread to enter an infinite loop and cease processing incoming packets.

New to v0.157 (12 July 2017)

Bundled with Heimdal 7.4 to address CVE-2017-11103 (Orpheus' Lyre puts Kerberos to sleep!)
"vos" support for volume quotas larger than 2TB.
"fs flushvolume" works
Fixed a bug that can result in a system panic during server capability testing

New to v0.150

AuriStorFS file server detection improvements

New to v0.149

rxkad encryption is enabled by default. Use "fs setcrypt off" to disable encryption when tokens are available.
Fix a bug in atomic operations on Sierra and El Capitan which could adversely impact Rx behavior.

New to v0.128

Extended attribute ._ files are automatically removed when the associated files are unlinked
Throughput improvements when sending data

New to v0.121

OSX Sierra support

New to v0.117

Cache file moved to a persistent location on local disk
AuriStor File System graphics
Improvements in Background token fetch functionality
Fixed a bug introduced in v0.44 that could result in an operating system crash when enumerating AFS directories containing Unicode file names (v0.106)
El Capitan security changes prevented Finder from deleting files and directories. As of v0.106, the AuriStor OSX client implements the required functionality to permit the DesktopHelperService to securely access the AFS cache as the user permitting Finder to delete files and directories.

Features:

Not vulnerable to OPENAFS-SA-2015-007.
Office 2011 can save to /afs.
Office 2016 can now save files to /afs.
OSX Finder and Preview can open executable documents without triggering a "Corrupted File" warning. .AI, .PDF, .TIFF, .JPG, .DOCX, .XLSX, .PPTX, and other structured documents that might contain scripts were impacted.
All file names are now stored to the file server using Unicode UTF-8 Normalization Form C which is compatible with Microsoft Windows.
All file names are converted to Unicode UTF-8 Normalization Form D for processing by OSX applications.

Known issues:

None

Windows Installer (64-bit)

Available to AuriStor File System Licensees
Please Contact Us for more information.

Windows Installer (32-bit)

Available to AuriStor File System Licensees
Please Contact Us for more information.

iOS Installer (iPhone)

COMING SOON

iOS Installer (iPad)

COMING SOON

Solaris Installer

Available to AuriStor File System Licensees
Please Contact Us for more information.

AuriStor File System Client Installers

RedHat Enterprise, CentOS, and Fedora Linux Repository installer

Release Notes

Installation Instructions

GPG Signing Key expired on 2 May 2025

New v2021.05-64 (22 Aug 2025)

v2021.05-63 (31 May 2025)

v2021.05-62 (24 April 2025)

New platform support

v2021.05-61 (31 March 2025)

v2021.05-60 (6 March 2025)

v2021.05-59 (13 February 2025)

v2021.05-57 (9 February 2025)

v2021.05-56 (7 February 2025)

v2021.05-55 (23 January 2025)

v2021.05-54 (19 January 2025)

v2021.05-53 (4 January 2025)

v2021.05-52 (5 December 2024)

v2021.05-51 (26 November 2024)

v2021.05-50 (23 November 2024)

v2021.05-49 (16 November 2024)

v2021.05-48 (12 November 2024)

v2021.05-47 (1 November 2024)

v2021.05-46 (28 October 2024)

v2021.05-45 (Not released)

v2021.05-44 (17 August 2024)

v2021.05-41 (26 June 2024)

v2021.05-40

v2021.05-39 (20 May 2024)

v2021.05-38 (29 February 2024)

v2021.05-37 (5 February 2024)

v2021.05-36 (10 January 2024)

New v2021.05-34 (21 December 2023)

New v2021.05-33 (27 November 2023)

v2021.05-32 (9 October 2023)

v2021.05-31 (25 September 2023)

v2021.05-30 (6 September 2023)

v2021.05-29 (26 June 2023)

v2021.05-28 (10 May 2023)

v2021.05-27 (1 May 2023)

v2021.05-26 (17 April 2023)

v2021.05-25 (28 December 2022)

AlmaLinux and Rocky Linux Repositories added (2 November 2022)

v2021.05-23 (4 October 2022)

v2021.05-22 (12 September 2022) and v2021.05-21 (6 September 2022)

New to v2021.05-20 (15 August 2022) and v2021.05-19 (13 August 2022)

New to v2021.05-18 (12 July 2022)

New to v2021.05-17 (16 May 2022)

New to v2021.05-16 (24 March 2022)

New to v2021.05-15 (24 January 2022)

New to v2021.05-14 (20 January 2022)

New to v2021.05-12 (7 October 2021)

New to v2021.05-9 (25 October 2021)

New to v2021.05-7 (22 August 2021)

New to v2021.05-3 (10 June 2021)

New to v2021.05 (31 May 2021)

New to v2021.04 (22 April 2021)

New to v0.209 (13 March 2021)

New to v0.200 (4 November 2020)

New to v0.197 (26 August 2020) and v0.198 (10 October 2020)

New to v0.195 (14 May 2020)

New to v0.194 (3 April 2020)

New to v0.192 (30 January 2020)

New to v0.191 (16 December 2019)

New to v0.190 (14 November 2019)

New to v0.189 (28 October 2019)

New to v0.188 (23 June 2019)

New to v0.186 (29 May 2019)

New to v0.184 (26 March 2019)

New to v0.179 and v0.180 (9 November 2018)

New to v0.170 (27 April 2018)

New to v0.168

New to v0.167

New to v0.164

New to v0.163

New to v0.160

New to v0.159

New to v0.157

New to v0.150

New to v0.147