Second mandatory service node upgrade

Oxen is announcing another Mandatory Service Node Upgrade (MSNU). Oxen 10.6.0 Wistful Wagyu will be released today (July 23 2024) with a 9 day mandatory upgrade period ending on August 1st 2024. Specifically after these dates Service Nodes not running the latest version of Oxen software will be decommissioned by the network: Wed Jul 31 23:30:00 2024 UTC Thu Aug 1 09:30:00 2024 Australia/Melbourne Wed Jul 31 19:30:00 2024 US/Eastern This upgrade further addresses network instability experienced by Session users in the last few weeks, adding to the fixes provided by the 10.5.0 release. Following the upgrade to 10.5.0, the reliability of one-on-one chats in Session was greatly improved, however, interactions with Communities and the File Server remained degraded.

Troubleshooting the problem

The team investigated the issues over the weekend and discovered a problem with the storage server's communication via HTTPS to non-service nodes. The storage server typically communicates with three external sources: SOGS (servers which host Communities), the Session file server, and the push notification server.

Resultantly, users experienced problems with file attachment uploads/downloads, Communities and the push notification server. This issue originated from two main sources:

1. Our implementation of requests from the storage server to external nodes could only handle a small number of simultaneous requests before the queue began to get backed up. Once the storage server became backed up with requests this would trigger further back ups, resulting in a vicious cycle where requests would end up in a queue for excessive periods of time.

2. A deadlock issue in libcpr, a C++ library used for sending those requests (see this issue) which would cause requests to be further delayed. 

The combination of these issues caused requests to frequently timeout and fail when sent to Communities, the Session file server and the push notification server.

To address this, we have refactored how these requests are made to make them asynchronous, which will allow us to use the available resources on all Service Nodes more efficiently, thus we can handle a higher volume of simultaneous requests. This change also ends up avoiding the part of cpr that could potentially deadlock entirely. These changes should resolve the issues with Session clients communicating reliably with Communities, the Session file server and push notification server (see this pull request for all the details).

The updated software was deployed on the OPTF's nodes and these servers were stress-tested using a script built by the team. During the tests, the request failure rate reduced from 95%+ (under stress) to just 0.008% from a sample of over 5.5 million requests.

Upgrading

Upgrading to the updated software is simple using your CLI.

Sync your repositories using the command

sudo apt update

Then install the update using the command

sudo apt upgrade

If you require any help, visit the the Oxen Service Nodes channel on Telegram or the Oxen Service Nodes channel on the Session Token Discord.