Feature ETP-4296: Add guide for diagnosing idle in transaction connections

docs/developer-guide/etendo-classic/how-to-guides/how-to-diagnose-idle-in-transaction-connections.md

@@ -0,0 +1,96 @@
+---
+title: How to Diagnose Idle in Transaction Connections
+tags:
+  - Connection Pool
+  - Troubleshooting
+  - Database
+  - PostgreSQL
+  - Hibernate
+  - DAL
+status: beta
+---
+
+# How to Diagnose Idle in Transaction Connections
+
+!!! example "IMPORTANT: THIS IS A BETA VERSION"
+    This page is under active development and may contain **unstable or incomplete features**. Use it **at your own risk**.
+
+## Overview
+
+Etendo serves most requests through `DalFilter`, which opens a Hibernate/DAL session at the start of the request and closes it once the response is sent. Custom servlets, processes, or webservice endpoints that call `OBDal.getInstance()` or open a `SessionHandler` session outside that lifecycle are responsible for closing the session themselves.
+
+A session that is never closed keeps its underlying JDBC connection checked out of the pool with an open transaction. PostgreSQL reports that connection as `idle in transaction`. Left unbounded, this blocks autovacuum on the tables the transaction touched, causes table bloat, and degrades database performance over time. Restarting Tomcat clears the pool and hides the symptom temporarily, which is why the underlying leak can go unnoticed for a while.
+
+## Detecting the Problem
+
+Query PostgreSQL for connections stuck in this state:
+
+```sql
+SELECT pid, now() - xact_start AS duration, state, LEFT(query, 80) AS query_snippet
+FROM pg_stat_activity
+WHERE state = 'idle in transaction';
+```
+
+A growing number of rows that never clears until Tomcat restarts indicates a session that is not being closed somewhere in the request path.
+
+## Finding the Leak
+
+Review custom code paths that open a DAL or Hibernate session directly instead of relying on `DalFilter`. This is common in servlets that extend `HttpServlet` or `HttpBaseServlet` directly, bypassing the standard filter chain. Each of these paths must close its session explicitly in a `finally` block:
+
+```java
+try {
+  // ... request processing that uses OBDal.getInstance() ...
+} finally {
+  OBDal.getInstance().commitAndClose();
+}
+```
+
+Or, when using `SessionHandler` directly:
+
+```java
+try {
+  // ... request processing ...
+} finally {
+  SessionHandler.getInstance().commitAndClose();
+}
+```
+
+To confirm which code path is responsible before changing anything, enable abandoned-connection logging. This logs the stack trace of where a connection was borrowed once it has been checked out longer than expected, without closing anything:
+
+```properties title="gradle.properties"
+db.pool.logAbandoned=true
+db.pool.suspectTimeout=<seconds>
+```
+
+Apply the change with:
+
+```bash
+./gradlew setup
+```
+
+!!! warning
+    Logging abandoned connections adds overhead to every connection borrow, because a stack trace has to be generated. Use it to diagnose the leak, then disable it once the responsible code path is identified.
+
+## Mitigating at the Pool Level
+
+While the code fix is developed and rolled out, the connection pool can be configured to forcibly reclaim connections that have been checked out too long. Add the following properties to `gradle.properties`:
+
+```properties title="gradle.properties"
+db.pool.removeAbandoned=true
+db.pool.removeAbandonedTimeout=<seconds>
+```
+
+Then apply the change:
+
+```bash
+./gradlew setup
+```
+
+See [How to Use an External Connection Pool](how-to-use-an-external-connection-pool.md#pool-configuration) for the full reference of pool configuration properties.
+
+!!! warning
+    Set `removeAbandonedTimeout` well above the longest legitimate transaction or background process duration in the environment. Any operation still running past that timeout has its connection reclaimed while in use, which corrupts that operation. This setting is a temporary safety net, not a substitute for closing the session in code.
+
+---
+
+This work is licensed under :material-creative-commons: :fontawesome-brands-creative-commons-by: :fontawesome-brands-creative-commons-sa: [CC BY-SA 2.5 ES](https://creativecommons.org/licenses/by-sa/2.5/es/){target="_blank"} by [Futit Services S.L](https://etendo.software){target="_blank"}.

docs/developer-guide/etendo-classic/how-to-guides/how-to-use-an-external-connection-pool.md

@@ -17,25 +17,90 @@ status: beta
 
 ## Overview
 
-By default, Etendo uses two connection pools:
+Etendo ships with the [Apache JDBC Connection Pool](https://github.com/etendosoftware/etendo_core/tree/main/modules_core/org.openbravo.apachejdbcconnectionpool){target="\_blank"} enabled by default, through the `org.openbravo.apachejdbcconnectionpool` core module. This module implements Etendo's `ExternalConnectionPool` abstraction on top of the [Apache Tomcat JDBC Connection Pool](https://tomcat.apache.org/tomcat-9.0-doc/jdbc-pool.html){target="\_blank"}. No installation step is required: `Openbravo.properties` already sets
 
-- Hibernate default connection pool for DAL-related queries 
-- [Apache DBCP](https://commons.apache.org/proper/commons-dbcp/){target="\_blank"} for the connections provided by the `ConnectionProviderImpl`. 
+```properties title="Openbravo.properties"
+db.externalPoolClassName=org.openbravo.apachejdbcconnectionpool.JdbcExternalConnectionPool
+```
 
-!!!info
-    It is possible to specify an external connection provider that Etendo will use to obtain the *JDBC connections*. For that, a module containing a subclass of `ExternalConnectionPool` needs to be installed, and the `db.externalPoolClassName` property has to be set in `gradle.properties` file.
+Because the pool is already active on every installation, the work described on this page is tuning its properties — see [Pool Configuration](#pool-configuration) below. Changing `db.externalPoolClassName` to a different class only becomes necessary to plug in a **custom** external connection pool implementation instead of the bundled one; see [How to Create an External Connection Pool](how-to-create-an-external-connection-pool.md) for that scenario.
+
+Pool properties are set in `gradle.properties`. After adding or changing any of them, apply the change with:
 
-## Example: Using the Apache JDBC Connection Pool
+```bash
+./gradlew setup
+```
 
-The [Apache JDBC Connection Pool](https://github.com/etendosoftware/etendo_core/tree/main/modules_core/org.openbravo.apachejdbcconnectionpool){target="\_blank"} module core provides an implementation of the Apache JDBC Connection Pool.
+This regenerates `Openbravo.properties`, which `JdbcExternalConnectionPool` reads at runtime to build the pool.
 
-The `db.externalPoolClassName` property has to be set in `gradle.properties`. This module implements the external connection pool class in the `org.openbravo.apachejdbcconnectionpool.JdbcExternalConnectionPool` class, so this line should be added to `gralde.properties`:
+## Pool Configuration
 
-``` title="Gradle.properties"
-db.externalPoolClassName=org.openbravo.apachejdbcconnectionpool.JdbcExternalConnectionPool
+A fresh Etendo installation ships with the following properties already set:
+
+```properties title="Openbravo.properties (generated)"
+db.pool.initialSize=1
+db.pool.minIdle=5
+db.pool.maxActive=10000
+db.pool.timeBetweenEvictionRunsMillis=60000
+db.pool.minEvictableIdleTimeMillis=120000
+db.pool.removeAbandoned=false
+db.pool.testOnBorrow=true
+db.pool.testWhileIdle=false
+db.pool.testOnReturn=false
+db.pool.validationQuery=SELECT 1 FROM DUAL
+db.pool.validationInterval=30000
+db.pool.jmxEnabled=false
 ```
 
-This module contains a configuration file template: `modules_core/org.openbravo.apachejdbcconnectionpool/config/connectionPool.properties.template`. In order to customize the JDBC connection pool properties this file has to be copied to `modules_core/org.openbravo.apachejdbcconnectionpool/config/connectionPool.properties`. The user can then configure the pool properties according to his needs. Hints about how to configure this properties can be found [here](https://tomcat.apache.org/){target="\_blank"}.
+Override any of them by setting the corresponding property in `gradle.properties`:
+
+| Property | Description | Default |
+| --- | --- | --- |
+| `db.pool.initialSize` | Connections established when the pool starts. Lowered automatically if it exceeds `db.pool.maxActive`. | `1` |
+| `db.pool.minIdle` | Minimum established connections kept in the pool at all times. The idle pool does not shrink below this value during an eviction run, but it can still drop lower if `db.pool.validationQuery` fails and connections are closed. | `5` |
+| `db.pool.maxActive` | Maximum active connections the pool can hand out at the same time. Kept high by default because capacity planning is delegated to the database; it should be at least as high as the database's own maximum connections. If lowered below that, `db.pool.maxWait` becomes relevant. | `10000` |
+| `db.pool.timeBetweenEvictionRunsMillis` | How often (ms) the sweeper thread checks idle and abandoned connections. See [How does the sweeper thread work?](#how-does-the-sweeper-thread-work). Should not be set below `1000`. | `60000` |
+| `db.pool.minEvictableIdleTimeMillis` | Minimum time (ms) a connection may sit idle before the sweeper evicts it. | `120000` |
+| `db.pool.removeAbandoned` | If `true`, connections held longer than `db.pool.removeAbandonedTimeout` are forcibly reclaimed. | `false` |
+| `db.pool.testOnBorrow` | Validates a connection before handing it out; drops and retries if invalid. Requires `db.pool.validationQuery` to be set. | `true` |
+| `db.pool.testOnReturn` | Validates a connection when it is returned to the pool. | `false` |
+| `db.pool.testWhileIdle` | Validates idle connections periodically. | `false` |
+| `db.pool.validationQuery` | SQL used to validate a connection. Must not throw an exception. Required for `testOnBorrow`, `testOnReturn`, and `testWhileIdle` to have any effect. Etendo's database creation scripts provision a `DUAL` compatibility table on PostgreSQL installations, so this query runs unmodified on both PostgreSQL and Oracle. | `SELECT 1 FROM DUAL` |
+| `db.pool.validationInterval` | Minimum milliseconds between validations of the same connection, to avoid redundant checks. | `30000` |
+| `db.pool.jmxEnabled` | Exposes pool metrics through JMX. | `false` |
+
+The pool also supports the properties below. Etendo does not set a default for any of them — when a property is not set in `gradle.properties`, the underlying [Apache Tomcat JDBC Connection Pool](https://tomcat.apache.org/tomcat-9.0-doc/jdbc-pool.html#Common_Attributes){target="\_blank"} default applies instead:
+
+| Property | Description |
+| --- | --- |
+| `db.pool.maxIdle` | Maximum idle connections kept in the pool when the sweeper is disabled. |
+| `db.pool.maxWait` | Milliseconds the pool waits for a connection to be returned before throwing an exception, once `db.pool.maxActive` has been reached. |
+| `db.pool.numTestsPerEvictionRun` | Number of connections examined in each sweeper run. |
+| `db.pool.removeAbandonedTimeout` | Seconds a connection can be checked out before it is considered abandoned. Only relevant when `db.pool.removeAbandoned=true`. |
+| `db.pool.testOnConnect` | Validates a connection right after it is physically created. |
+| `db.pool.validatorClassName` | Custom validator class used instead of `db.pool.validationQuery`. |
+| `db.pool.initSQL` | SQL executed once, right after a physical connection is created. |
+| `db.pool.defaultAutoCommit` | Default auto-commit state of connections returned by the pool. |
+| `db.pool.defaultReadOnly` | Default read-only state of connections returned by the pool. |
+| `db.pool.defaultTransactionIsolation` | Default transaction isolation level of connections returned by the pool. |
+| `db.pool.defaultCatalog` | Default catalog of connections returned by the pool. |
+| `db.pool.connectionProperties` | Extra driver-specific connection properties, as a semicolon-separated list of `name=value` pairs. |
+| `db.pool.accessToUnderlyingConnectionAllowed` | Allows retrieving the underlying physical connection through the pooled connection wrapper. |
+| `db.pool.logAbandoned` | Logs the stack trace of where a connection was borrowed once it has been checked out longer than `db.pool.suspectTimeout`. Adds overhead to every borrow, since a stack trace has to be generated. |
+| `db.pool.suspectTimeout` | Seconds a connection can be checked out before it is logged as suspect. Only relevant when `db.pool.logAbandoned=true`. Independent from `db.pool.removeAbandoned` — a suspect connection is only logged, not reclaimed. |
+| `db.pool.name` | Name assigned to the pool, useful to tell pools apart when several are configured. |
+
+!!!info
+    Any of these properties can be scoped to a specific named pool — for example the read-only pool — by inserting the pool name after `db.`, e.g. `db.readonly.pool.maxActive`. A pool-specific value takes precedence over the default `db.pool.*` value for that pool; a pool that does not define its own value falls back to the default.
+
+### How does the sweeper thread work?
+
+The sweeper is the background thread that runs every `db.pool.timeBetweenEvictionRunsMillis` milliseconds to validate idle connections and check for abandoned ones. Whether it is enabled changes how the idle pool behaves:
+
+- **Sweeper disabled**: if the idle pool grows larger than `db.pool.maxIdle`, a connection is closed as soon as it is returned to the pool instead of being kept idle.
+- **Sweeper enabled**: the number of idle connections can grow beyond `db.pool.maxIdle`, but shrinks back down to `db.pool.minIdle` once a connection has been idle longer than `db.pool.minEvictableIdleTimeMillis`.
+
+The full list of configurable Tomcat JDBC Connection Pool attributes is available in the [Apache Tomcat documentation](https://tomcat.apache.org/tomcat-9.0-doc/jdbc-pool.html#Common_Attributes){target="\_blank"}, along with [guidance on tuning the pool for high-concurrency environments](https://www.tomcatexpert.com/blog/2010/04/01/configuring-jdbc-pool-high-concurrency){target="\_blank"}.
 
 ---
-This work is a derivative of [How to Use an External Connection Pool](http://wiki.openbravo.com/wiki/How_to_Use_an_External_Connection_Pool){target="\_blank"} by [Openbravo Wiki](http://wiki.openbravo.com/wiki/Welcome_to_Openbravo){target="\_blank"}, used under [CC BY-SA 2.5 ES](https://creativecommons.org/licenses/by-sa/2.5/es/){target="\_blank"}. This work is licensed under [CC BY-SA 2.5](https://creativecommons.org/licenses/by-sa/2.5/){target="\_blank"} by [Etendo](https://etendo.software){target="\_blank"}.
+This work is a derivative of [How to Use an External Connection Pool](http://wiki.openbravo.com/wiki/How_to_Use_an_External_Connection_Pool){target="\_blank"} by [Openbravo Wiki](http://wiki.openbravo.com/wiki/Welcome_to_Openbravo){target="\_blank"}, used under [CC BY-SA 2.5 ES](https://creativecommons.org/licenses/by-sa/2.5/es/){target="\_blank"}. This work is licensed under [CC BY-SA 2.5](https://creativecommons.org/licenses/by-sa/2.5/){target="\_blank"} by [Etendo](https://etendo.software){target="\_blank"}.

docs/es/developer-guide/etendo-classic/how-to-guides/how-to-diagnose-idle-in-transaction-connections.md

@@ -0,0 +1,96 @@
+---
+title: Cómo diagnosticar conexiones en estado idle in transaction
+tags:
+  - Pool de conexiones
+  - Solución de problemas
+  - Base de datos
+  - PostgreSQL
+  - Hibernate
+  - DAL
+status: beta
+---
+
+# Cómo diagnosticar conexiones en estado idle in transaction { #how-to-diagnose-idle-in-transaction-connections }
+
+!!! example "IMPORTANTE: ESTA ES UNA VERSIÓN BETA"
+    Esta página está en desarrollo activo y puede contener **funcionalidades inestables o incompletas**. Úsela **bajo su propia responsabilidad**.
+
+## Visión general { #overview }
+
+Etendo atiende la mayoría de las solicitudes a través de `DalFilter`, que abre una sesión de Hibernate/DAL al inicio de la solicitud y la cierra una vez enviada la respuesta. Los servlets, procesos o endpoints de servicios web personalizados que llaman a `OBDal.getInstance()` o abren una sesión de `SessionHandler` fuera de ese ciclo de vida son responsables de cerrar la sesión ellos mismos.
+
+Una sesión que nunca se cierra mantiene su conexión JDBC subyacente retirada del pool con una transacción abierta. PostgreSQL informa esa conexión como `idle in transaction`. Si no se controla, esto bloquea el autovacuum en las tablas que tocó la transacción, provoca hinchazón de las tablas y degrada el rendimiento de la base de datos con el tiempo. Reiniciar Tomcat vacía el pool y oculta el síntoma temporalmente, por lo que la fuga subyacente puede pasar inadvertida durante un tiempo.
+
+## Detección del problema { #detecting-the-problem }
+
+Consulte PostgreSQL para buscar conexiones atascadas en este estado:
+
+```sql
+SELECT pid, now() - xact_start AS duration, state, LEFT(query, 80) AS query_snippet
+FROM pg_stat_activity
+WHERE state = 'idle in transaction';
+```
+
+Un número creciente de filas que nunca se despeja hasta que se reinicia Tomcat indica que, en algún punto del recorrido de la solicitud, no se está cerrando una sesión.
+
+## Cómo encontrar la fuga { #finding-the-leak }
+
+Revise los caminos de código personalizados que abren una sesión DAL o de Hibernate directamente en lugar de depender de `DalFilter`. Esto es habitual en servlets que extienden directamente `HttpServlet` o `HttpBaseServlet`, evitando la cadena de filtros estándar. Cada uno de estos caminos debe cerrar su sesión explícitamente en un bloque `finally`:
+
+```java
+try {
+  // ... procesamiento de la solicitud que usa OBDal.getInstance() ...
+} finally {
+  OBDal.getInstance().commitAndClose();
+}
+```
+
+O, cuando se usa `SessionHandler` directamente:
+
+```java
+try {
+  // ... procesamiento de la solicitud ...
+} finally {
+  SessionHandler.getInstance().commitAndClose();
+}
+```
+
+Para confirmar qué camino de código es responsable antes de cambiar nada, habilite el registro de conexiones abandonadas. Esto registra la traza de pila de dónde se tomó prestada una conexión una vez que ha estado retirada más tiempo del esperado, sin cerrar nada:
+
+```properties title="gradle.properties"
+db.pool.logAbandoned=true
+db.pool.suspectTimeout=<segundos>
+```
+
+Aplique el cambio con:
+
+```bash
+./gradlew setup
+```
+
+!!! warning
+    Registrar las conexiones abandonadas añade sobrecarga a cada solicitud de conexión, porque se debe generar una traza de pila. Utilícelo para diagnosticar la fuga y luego deshabilítelo una vez identificado el camino de código responsable.
+
+## Mitigación a nivel de pool { #mitigating-at-the-pool-level }
+
+Mientras se desarrolla e implementa la corrección de código, el pool de conexiones puede configurarse para recuperar por la fuerza las conexiones que se han retirado durante demasiado tiempo. Añada las siguientes propiedades a `gradle.properties`:
+
+```properties title="gradle.properties"
+db.pool.removeAbandoned=true
+db.pool.removeAbandonedTimeout=<segundos>
+```
+
+Luego aplique el cambio:
+
+```bash
+./gradlew setup
+```
+
+Consulte [Cómo usar un pool de conexiones externo](how-to-use-an-external-connection-pool.md#pool-configuration) para ver la referencia completa de las propiedades de configuración del pool.
+
+!!! warning
+    Establezca `removeAbandonedTimeout` con un margen amplio por encima de la duración de la transacción o el proceso en segundo plano legítimo más largo del entorno. Cualquier operación que siga en ejecución más allá de ese tiempo de espera tendrá su conexión recuperada mientras está en uso, lo que corrompe esa operación. Este ajuste es una red de seguridad temporal, no un sustituto de cerrar la sesión en el código.
+
+---
+
+This work is licensed under :material-creative-commons: :fontawesome-brands-creative-commons-by: :fontawesome-brands-creative-commons-sa: [CC BY-SA 2.5 ES](https://creativecommons.org/licenses/by-sa/2.5/es/){target="_blank"} by [Futit Services S.L](https://etendo.software){target="_blank"}.