DOC-13655 document kernel tcp_mem settings.

modules/ROOT/nav.adoc

@@ -202,6 +202,7 @@ include::third-party:partial$nav.adoc[]
  ** xref:install:synchronize-clocks-using-ntp.adoc[Clock Synchronization Requirements]
  ** xref:install:thp-disable.adoc[Disable THP]
  ** xref:install:install-swap-space.adoc[Configure Kernel Swappiness]
+ ** xref:install:tcp_mem_settings.adoc[] 
  ** xref:install:install-security-bp.adoc[Security Considerations]
  ** xref:install:server-processes.adoc[Couchbase Server Process List]
  ** xref:install:best-practices-vm.adoc[VM and Container Guidelines]

modules/install/pages/tcp_mem_settings.adoc

@@ -0,0 +1,229 @@
+= Linux Kernel TCP/IP Memory Settings
+:description: pass:q[The Linux kernel has a global parameter named `tcp_mem` that limits the TCP/IP stack's RAM use.]
+:page-topic-type: concept
+:stem: asciimath
+
+[abstract]
+{description}
+The goal of this setting is to prevent the TCP/IP stack from consuming so much memory under heavy network use that it interferes with other workloads.
+The Linux Kernel sets default values for `tcp_mem` based on the total amount of RAM and other factors in the system.
+However, these defaults may not be suitable for Couchbase Server workloads. 
+You should verify and, if necessary, adjust this setting to prevent potential networking issues.
+
+[#about]
+== About `tcp_mem`
+
+The Linux kernel's `tcp_mem` setting is a trio of integers that sets thresholds for the TCP/IP stack's memory use.
+You can view the current `tcp_mem` settings by running the following command:
+
+[source,sh]
+----
+cat /proc/sys/net/ipv4/tcp_mem
+----
+
+NOTE: Even though `tcp_mem` appears within the `ipv4` directory, it applies to both IPv4 and IPv6 traffic.
+There's no separate `tcp_mem` setting for IPv6.
+
+The command returns 3 integer values representing memory thresholds in pages.
+The following output is from a system with 8{nbsp}GB of RAM:
+
+[source,console]
+----
+93933	125246	187866
+----
+
+The 3 values in `tcp_mem` are:
+
+* The minimum threshold: When the total memory used by all TCP sockets is less than this value, the TCP/IP stack stops taking steps to reduce its memory use.
+* The pressure threshold: When the TCP/IP stack's memory use exceeds this value, the TCP/IP stack starts taking steps reduce its memory use.
+These steps it takes include:
++
+--
+** Capping the size of TCP/IP windows. 
+These windows control how much data the TCP/IP stack can transmit at a time.
+** Freeing excess memory from TCP socket buffers and aggressively removing buffers of sockets that are closing.
+** Limiting the amount of memory allocated for incoming packets.  
+--
+
++
+These measures can affect network performance.
+Symptoms of the TCP/IP memory use being under pressure include increased latency and limited throughput.
+The TCP/IP stack continues to apply these measures until its memory use drops to less than the minimum threshold.
+
+* The maximum threshold: If the TCP/IP stack's memory use reaches this limit, the kernel prevents it from allocating more memory. 
+Hitting this limit results in dropped packets and refused connections.
+The kernel also begins logging Out Of Memory (OOM) errors to the system log.
+For example, you may see the following message in the `dmesg` log:
+
++
+[source,console]
+----
+TCP: out of memory -- consider tuning tcp_mem
+----
+
+[IMPORTANT] 
+====
+The values returned by the command (and the values you use to set these thresholds) represent the limits in number of memory pages, not bytes.
+The size of a Linux kernel memory page is typically 4{nbsp}KB, but it can vary depending on the system architecture and configuration.
+You can verify the memory page size on your system by running the following command:
+
+[source,sh]
+----
+getconf PAGE_SIZE
+----
+
+It returns the number of bytes in a memory page (for example, `4096` for 4 KB pages).
+To convert the `tcp_mem` values from pages to bytes, multiply each value by the page size.
+For example, if the page size is 4{nbsp}KB (4096 bytes), the above `tcp_mem` values in bytes are `384749568 513007616 769499136`.
+====
+
+== Recommended `tcp_mem` Settings
+
+Couchbase Server relies on network communication for cluster operations, data replication, and client interactions.
+It needs low-latency network access for vital features such as the node heartbeat mechanism that detects node failures and triggers failovers.
+
+The default `tcp_mem` settings in some Linux distributions may not be sufficient for Couchbase Server workloads, especially under high network load.
+The default settings the kernel uses take into account different factors.
+Verify the `tcp_mem` value your system sets using the command shown in <<about>>.
+
+To prevent potential networking issues, the `tcp_mem` values should allow the TCP/IP stack to use between 5% and 10% of a node's total RAM.
+If the automatically set value is lower than this range, you should adjust the `tcp_mem` settings.
+Decide whether to allow the kernel to allocate up to 5% or 10% of the total RAM.
+You use this value for the maximum threshold.
+Set the other 2 thresholds relative to the maximum:
+
+* Set the minimum threshold to 80% of the maximum threshold.
+* Set the pressure threshold to 90% of the maximum threshold.
+
+For example, suppose your nodes have 64{nbsp}GB of RAM and 4{nbsp}KB page size.
+Also suppose you want to let the TCP/IP stack use up to 5% of the system RAM.
+Then you can follow these steps to calculate the recommended `tcp_mem` settings:
+
+. Calculate the total number of bytes in the node's RAM: stem:[64 xx 1024 xx 1024 xx 1024 = 68,719,476,736] bytes.
+. Calculate the total number of memory pages in the system by dividing the total bytes by the page size: stem:["68,719,476,736" / "4,096" = 16,777,216] pages.
+. Calculate the maximum threshold as 5% of the total memory pages: stem:[16,777,216 xx 0.05 = 838,860.8] pages.
+Round the result to the nearest whole number: stem:[838,861] pages. 
+. Calculate the pressure threshold as 90% of the maximum threshold: stem:[838,861 xx 0.90 = 754,974.9] pages.
+Round up to the nearest whole number: stem:[754,975] pages.
+. Calculate the minimum threshold as 80% of the maximum threshold: stem:[838,861 xx 0.80 = 671,088.8] pages.
+Round up to the nearest whole number: stem:[671,089] pages.
+
+As a result, the recommended `tcp_mem` settings for this example are:
+
+* Minimum threshold: 671089
+* Pressure threshold: 754975
+* Maximum threshold: 838861
+
+The calculations for nodes with the same RAM and page size for a 10% memory limit are:
+
+* Minimum threshold: 1342178
+* Pressure threshold: 1509950
+* Maximum threshold: 1677722
+
+== Setting `tcp_mem`
+
+You can change the `tcp_mem` settings temporarily or permanently.
+
+=== Temporarily Set `tcp_mem`
+
+You can set `tcp_mem` temporarily (until the next reboot), using the `sysctl` command as root. 
+You may want to temporarily set the value to test the new settings or to resolve an urgent network issue.
+If the new settings resolve your issue, you can then <<permanent,make them permanent>>.
+
+To immediately change `tcp_mem`, run following command as root, replacing the example values with your calculated values:
+
+[source,sh
+----
+sysctl -w net.ipv4.tcp_mem="<minimum> <pressure> <maximum>"
+----
+
+For example, to set the `tcp_mem` values calculated in the previous section for a 5% memory limit, run the following command:
+
+[source,sh]
+----
+sysctl -w net.ipv4.tcp_mem="671089 754975 838861"
+----
+
+If successful, the command returns the updated setting:
+
+[source,console]
+----
+net.ipv4.tcp_mem = 671089 754975 838861
+----
+
+You can also verify the new settings by examining the `/proc/sys/net/ipv4/tcp_mem` file:
+
+[source,sh]
+----
+cat /proc/sys/net/ipv4/tcp_mem
+----
+
+The result of the command shows the current values:
+
+[source,console]
+----
+671089	754975	838861
+----
+
+[#permanent]
+=== Permanently Set `tcp_mem`
+
+To set `tcp_mem` permanently, you add or modify a setting in the Linux `sysctl` configuration. 
+The exact file to edit depends on your Linux distribution.
+Traditionally, you would add the setting to the `/etc/sysctl.conf` file.
+However, recent Linux distributions have deprecated the use of this file in favor of files containing individual settings in the `/etc/sysctl.d/` directory.
+Consult your Linux distribution's documentation for the recommended way to set `sysctl` parameters permanently.
+
+Once you have determined the file to edit, add or modify the following line. 
+Replace the example values with your calculated values:
+
+[source,sh]
+----
+# Adjust tcp_mem settings for Couchbase Server
+net.ipv4.tcp_mem = <minimum> <pressure> <maximum>
+----
+
+For example, to set the `tcp_mem` values calculated in the previous section for a 5% memory limit on a Debian 13 system, you could follow these steps:
+
+. Using your preferred text editor, create a new file named `/etc/sysctl.d/99-couchbase-tcp-mem.conf`.
+. Add the following line to the file:
+
++
+[source,sh]
+----
+# Adjust tcp_mem settings for Couchbase Server
+net.ipv4.tcp_mem = 671089 754975 838861
+----
+
+. Save the file and exit the text editor.
+
+. To apply the new settings without rebooting, run the following command as root:
+
++
+[source,sh]
+----
+sysctl --system
+----
+
++
+This command reloads all `sysctl` settings from the configuration files.
+It prints all the kernel settings, including the updated `net.ipv4.tcp_mem` setting (shown in a truncated form here):
+
++
+[source,console]
+----
+. . .
+net.ipv4.tcp_max_syn_backlog = 512
+net.ipv4.tcp_max_tw_buckets = 32768
+net.ipv4.tcp_mem = 671089 754975 838861
+net.ipv4.tcp_migrate_req = 0
+net.ipv4.tcp_min_rtt_wlen = 300
+. . .
+----
+
+TIP: If you notice that your configuration settings are not having an effect, search for any conflicting settings in other `sysctl` configuration files.
+Conflicts can happen if multiple files set the same parameter with different values.
+The last configuration file that Linux processes takes precedence.
+For example, if both `/etc/sysctl.conf` and a file in `/etc/sysctl.d/` set `net.ipv4.tcp_mem`, the kernel may use the value from `/etc/sysctl.conf`.
+This settings takes precedence because some Linux distributions process that file last.
+

modules/manage/pages/troubleshoot/common-errors.adoc

@@ -233,3 +233,24 @@ sudo echo '' >> /etc/sysctl.conf
 Make sure that you either have or modify your process that builds your OSs to do this.
 This is especially critical for public/private clouds where it is so easy to bring up new instances.
 You need to make this part of your build process for a Couchbase node.
+
+[#tcp_mem]
+Kernel TCP/IP memory settings::
+The Linux kernel has a global parameter named `tcp_mem` that sets limits on the TCP/IP stack's memory use.
+With an intense network workload, the TCP/IP stack could reach 1 of the thresholds defined by this setting.
+When it reaches the pressure threshold, the TCP/IP stack starts taking steps to reduce its memory use.
+These steps could cause the node to experience higher network latency and limited throughput.
+
++
+If the TCP/IP stack's memory use continues to increase to the maximum threshold, the node could start dropping packets or refusing connections.
+In addition, the kernel logs Out Of Memory (OOM) errors to the system logs, such as:
+
++
+[source,console]
+----
+TCP: out of memory -- consider tuning tcp_mem
+----
+
++
+To prevent or resolve these issues, you may need to adjust the `tcp_mem` settings.
+See xref:install:tcp_mem_settings.adoc[] for more information.