Skip to content

Feat/read replica for geocoding #305

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
joaquimds merged 2 commits into from
Feb 5, 2026
Merged

Feat/read replica for geocoding #305

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
85e947a
feat: add read replica for expensive geocode query
joaquimds Feb 4, 2026
a6c9bb8
feat: add area (area_set_id, geography) combined index for improved g…
joaquimds Feb 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 14 additions & 0 deletions migrations/1770221306067_area_set_id_geography_index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
/* eslint-disable @typescript-eslint/no-explicit-any */
import { type Kysely, sql } from "kysely";

export async function up(db: Kysely<any>): Promise<void> {
await sql`CREATE EXTENSION IF NOT EXISTS btree_gist`.execute(db);
await sql`CREATE INDEX area_area_set_id_geography_gist ON area USING GIST (area_set_id, geography)`.execute(
db,
);
}

export async function down(db: Kysely<any>): Promise<void> {
await sql`DROP INDEX IF EXISTS area_area_set_id_geography_gist`.execute(db);
await sql`DROP EXTENSION IF EXISTS btree_gist`.execute(db);
}
Comment on lines +13 to +14
Copy link

Copilot AI Feb 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dropping the btree_gist extension in the down migration is potentially unsafe. If other GIST indexes or database objects depend on this extension, the DROP EXTENSION command will fail. PostgreSQL extensions are shared across the database, not tied to individual indexes. Consider removing this line from the down migration, as the extension may be needed by other parts of the database or future migrations. The index removal on line 12 is sufficient for the down migration.

Suggested change
await sql`DROP EXTENSION IF EXISTS btree_gist`.execute(db);
}
}

Copilot uses AI. Check for mistakes.
9 changes: 7 additions & 2 deletions src/server/commands/ensureOrganisationMap.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,11 @@ import {
} from "@/server/repositories/MapView";
import { upsertOrganisation } from "@/server/repositories/Organisation";
import { AreaSetCode, AreaSetGroupCode } from "../models/AreaSet";
import { ColorScheme, MapStyleName } from "../models/MapView";
import {
ColorScheme,
DEFAULT_CALCULATION_TYPE,
MapStyleName,
} from "../models/MapView";
import { countDataRecordsForDataSource } from "../repositories/DataRecord";
import type { MapView } from "../models/MapView";
import type { DataSource } from "@/server/models/DataSource";
Expand Down Expand Up @@ -87,11 +91,12 @@ const ensureOrganisationMap = async (orgId: string): Promise<Map> => {
areaDataColumn: "Lab",
areaDataSourceId: electionResultsDataSource.id,
areaSetGroupCode: AreaSetGroupCode.WMC24,
calculationType: null,
calculationType: DEFAULT_CALCULATION_TYPE,
Copy link

Copilot AI Feb 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change from null to DEFAULT_CALCULATION_TYPE appears unrelated to the read replica feature described in the PR title. While this may be a valid improvement, including unrelated changes in a feature PR can make code review more difficult and complicates the git history. Consider moving this to a separate PR or documenting it in the PR description.

Copilot uses AI. Check for mistakes.
colorScheme: ColorScheme.GreenYellowRed,
mapStyleName: MapStyleName.Light,
reverseColorScheme: false,
showBoundaryOutline: true,
showChoropleth: true,
Copy link

Copilot AI Feb 4, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The addition of showChoropleth appears unrelated to the read replica feature described in the PR title. While this may be a valid improvement, including unrelated changes in a feature PR can make code review more difficult and complicates the git history. Consider moving this to a separate PR or documenting it in the PR description.

Copilot uses AI. Check for mistakes.
showLabels: true,
showLocations: true,
showMembers: true,
Expand Down
5 changes: 3 additions & 2 deletions src/server/repositories/Area.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
import { sql } from "kysely";
import { db } from "@/server/services/database";
import { db, dbRead } from "@/server/services/database";
import type { Area } from "@/server/models/Area";
import type { AreaSetCode } from "@/server/models/AreaSet";
import type { Database } from "@/server/services/database";
Expand Down Expand Up @@ -82,7 +82,8 @@ export async function findAreasByPoint({
excludeAreaSetCode?: AreaSetCode | null | undefined;
includeAreaSetCode?: AreaSetCode | null | undefined;
}): Promise<AreaWithAreaSetCode[]> {
let query = db
// Use the read replica for this expensive read query
let query = dbRead
.selectFrom("area")
.innerJoin("areaSet", "area.areaSetId", "areaSet.id");
if (excludeAreaSetCode) {
Expand Down
32 changes: 25 additions & 7 deletions src/server/services/database/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,11 +25,23 @@ export const pool = new Pool({
max: Number(process.env.DATABASE_POOL_SIZE) || undefined,
});

// Set up read replica pool with graceful fallback
const readReplicaPool = new Pool({
connectionString:
process.env.DATABASE_READ_REPLICA_URL || process.env.DATABASE_URL,
max: Number(process.env.DATABASE_POOL_SIZE) || undefined,
});

const dialect = new PostgresDialect({
cursor: Cursor,
pool,
});

const readReplicaDialect = new PostgresDialect({
cursor: Cursor,
pool: readReplicaPool,
});

export interface Database {
airtableWebhook: AirtableWebhookTable;
area: AreaTable;
Expand All @@ -49,14 +61,20 @@ export interface Database {
"pgboss.job": JobTable;
}

const sharedPlugins = [
new CamelCasePlugin({ maintainNestedObjectKeys: true }),
new PointPlugin(),
new JSONPlugin(),
];

export const db = new Kysely<Database>({
dialect,
plugins: [
// Database `field_name` to TypeScript `fieldName`.
// `maintainNestedObjectKeys` prevents `data_record.json` being mangled
new CamelCasePlugin({ maintainNestedObjectKeys: true }),
new PointPlugin(),
new JSONPlugin(),
],
plugins: sharedPlugins,
log: ["error"],
});

export const dbRead = new Kysely<Database>({
dialect: readReplicaDialect,
plugins: sharedPlugins,
log: ["error"],
});
Loading