Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 33 additions & 14 deletions pgdog/src/frontend/client/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -531,43 +531,46 @@ impl Client {

/// Handle client messages.
async fn client_messages(&mut self, query_engine: &mut QueryEngine) -> Result<(), Error> {
// Check maintenance mode.
if !self.in_transaction()
&& !self.admin
&& let Some(waiter) = maintenance_mode::waiter(&self.database)
{
let state = query_engine.get_state();
query_engine.set_state(State::Waiting);
waiter.await;
query_engine.set_state(state);
}

// If client sent multiple requests, split them up and execute individually.
let spliced = self.client_request.spliced()?;
if spliced.is_empty() {
// Execute the original request as-is.
let mut context = QueryEngineContext::new(self);
query_engine.handle(&mut context).await?;
self.transaction = context.transaction();
} else {
// Execute the spliced requests one at a time.
let total = spliced.len();
let mut reqs = spliced.into_iter().enumerate();

// Pipeliend requests are executed inside the same Postgres transaction.
self.transaction.get_or_insert(TransactionType::Implicit);

while let Some((num, mut req)) = reqs.next() {
debug!("processing spliced request {}/{}", num + 1, total);

let mut context = QueryEngineContext::new(self).spliced(&mut req, reqs.len());
query_engine.handle(&mut context).await?;

// Save transaction state after each request.
self.transaction = context.transaction();

// If pipeline is aborted due to error, skip to Sync to complete the pipeline.
// Postgres ignores all commands after an error until it receives Sync.
// Postgres ignores all commands after an error until it receives a Sync.
if query_engine.out_of_sync() && !req.is_sync_only() {
debug!("pipeline aborted, skipping to Sync");

for (_, mut next_req) in reqs.by_ref() {
// The splicer (called above) leaves
// the Sync in its own request. This is by design
// to handle this exact scenario.
if next_req.is_sync_only() {
debug!("processing Sync to complete aborted pipeline");

let mut ctx = QueryEngineContext::new(self).spliced(&mut next_req, 0);
query_engine.handle(&mut ctx).await?;
self.transaction = ctx.transaction();

break;
}
}
Expand All @@ -582,13 +585,29 @@ impl Client {
Ok(())
}

/// Buffer extended protocol messages until client requests a sync.
/// Buffer protocol messages until client requests a sync or a flush.
///
/// This ensures we don't check out a connection from the pool until the client
/// sent a complete request.
/// sends a complete request.
///
async fn buffer(&mut self, state: State) -> Result<BufferEvent, Error> {
self.client_request.clear();

// Check maintenance mode.
//
// This will pause the client until the maintenance mode is off
// without checking any timeouts or reading data off of the socket
Comment on lines +598 to +599

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the buffer future is created once during select! in loop and in order to it to be recreated we should proceed to the next iteration of the loop.

that means we can get the following flow:

  1. client connects and stays idle
  2. no maintenance mode so we don't wait and go straight to buffer read
  3. we enable maintenance
  4. the idle client runs some command and we execute them without checks for maintenance

yes, other branches in select! could recreate the future, but that's not reliable.

So, moving the check upfront could it make it worse, because the check actually moved to the connection time whilst previously it was at execution time, when we actually got some commands.

I think we really need integration tests here to verify the behavior.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh wow. That's such a footgun...really good catch.

// so if the client disconnects while we wait, we won't send
// its pending query to the database.
Comment on lines +600 to +601

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess the socket data will still be in the kernel buffer and we'll still read and try to execute as soon as we read proper spliced part. The query will be executed then and we'll probably get an error when we write.

that's why another pr for the issue was trying to always read the stream to catch the EOF to understand if the client is disconnected during maintenance.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see that now.

if !self.in_transaction()
&& !self.admin
&& let Some(waiter) = maintenance_mode::waiter(&self.database)
{
self.comms.set_state(State::Waiting);
waiter.await;
self.comms.set_state(state);
}

// Only start timer once we receive the first message.
let mut timer = None;

Expand Down
39 changes: 34 additions & 5 deletions pgdog/src/frontend/comms.rs
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ use tokio_util::task::TaskTracker;

use crate::net::Parameters;
use crate::net::messages::{BackendKeyData, FrontendPid};
use crate::state::State;

use super::{ConnectedClient, Stats};

Expand Down Expand Up @@ -155,23 +156,51 @@ impl Deref for ClientComms {
}

impl ClientComms {
pub fn disconnect(&self) {
/// Client has disconnected, remove it from the global state.
pub(crate) fn disconnect(&self) {
self.comms.disconnect(self.id);
}

pub fn update_stats(&self, stats: Stats) {
/// Update global client stats with the latest stats the caller knows about.
/// The caller must maintain up-to-date stats, or this will overwrite with bad
/// info.
pub(crate) fn update_stats(&self, stats: Stats) {
self.comms.update_stats(self.id, stats);
}

pub fn new(id: FrontendPid) -> Self {
/// Update the client state quickly.
///
/// Use only when you're not updating other stats. If updating other stats,
/// use [`Self::update_stats`] instead.
///
pub(crate) fn set_state(&self, state: State) {
if let Some(mut client) = self.comms.global.clients.get_mut(&self.id) {
client.stats.state = state;
}
}

/// Create new comms reference for a new client.
///
/// Makes it easy to update globals without constantly reaching for the key.
pub(crate) fn new(id: FrontendPid) -> Self {
Self { id, comms: comms() }
}

pub fn connect(&self, key: BackendKeyData, addr: SocketAddr, params: &Parameters) {
/// Mark the client as connected to the proxy
/// and save some of its identifying information.
///
/// # Parameters
///
/// - `key`: The query cancellation key we gave to the client.
/// - `addr`: The client's IP address.
/// - `params`: The client's startup parameters.
///
pub(crate) fn connect(&self, key: BackendKeyData, addr: SocketAddr, params: &Parameters) {
self.comms.connect(key, addr, params)
}

pub fn update_params(&self, params: &Parameters) {
/// Update the params the client has in its session state.
pub(crate) fn update_params(&self, params: &Parameters) {
self.comms.update_params(self.id, params.clone());
}
}
Expand Down
Loading