fsyncDATA

Member sign in
Join

fsyncDATA

Apply or request an invitation

Membership is earned on evidence, not titles. Tell us what you run, how you run it, and one problem you have solved in production. A person reads every application, and a guild steward reads every production account in full.

There are three ways in. Practitioners apply to a Guild. Leaders request a Circle invitation. Anyone may nominate a peer. All three are free, personal and bound by the charter. Allow about ten minutes, and have your engine versions and HA/DR topology to hand.

Go to the form
What stewards look for

2

working days to a first reply from a person, never an autoresponder

10

engine guilds a practitioner can join, most members join two or three

28

engines and cloud database services with a guild that runs them

₹0

fee, in either tier, now or later. The price is one contribution a year

Three paths, one standard

Each path asks for different evidence. The form adapts once you choose: you will only see the questions for your path.

Practitioners · uncapped

Apply to a Guild

For principal and staff database architects, data architects, DBAs, DBREs and database engineers. We ask for the engines and versions you run, how they are deployed, your scale, your HA/DR topology with tested RPO/RTO, and either a vouching member or a 300-word production account.

Leaders · by nomination

Request a Circle invitation

For CDOs, CDAOs, chief data architects, CIOs, CTOs, GCC site leaders and heads of data platform. We ask for the estate you are accountable for, the size of the organisation you lead, and one decision you expect to face in the next twelve months.

Anyone

Nominate a peer

Tell us who they are, what they run, and why the room would be better with them in it. We contact them directly and name you only with your permission.

What stewards look for

Stewards are practitioners who run the same engines you do. They are not scoring seniority; they are looking for evidence that you operate production and can explain why it behaves the way it does.

SignalStrong evidenceWeak evidence
Engine depthNamed engines with major and minor versions; internals referenced precisely (vacuum horizon, InnoDB redo and flushing, plan regressions in Query Store, MergeTree merges)“Experienced in all major databases”
ScaleInstance counts, data volume and peak load with units: 18k write TPS, 40 TB, 2 TB/day ingest“Large-scale, high-performance systems”
DiagnosisThe metrics, system views or logs that found the cause: pg_stat_activity, performance_schema, sys.dm_exec_requests, AWR/ASH, system.query_log“Identified and resolved performance issues”
ReliabilityRPO and RTO targets with the result and date of the last real failover or restore drill“Ensured high availability”
JudgmentA rollback path, the trade-off you rejected, and what you changed so it would not recurA fix with no mention of risk
ContributionA concrete offer: a guild session, a post-mortem, peer-bench answers, the benchmark“Happy to help”

None of this requires naming your employer. Engine, version and numbers are enough.

Anatomy of a strong production account

If no member can vouch for you, write the account you would file after a real incident on an estate you operate. We read it the way a principal data architect reads a post-mortem in full-stack database infrastructure operations: across the engine, the storage and replication layers, the HA/DR topology, the managed-service control plane and the workload that triggered it. Name the engine, the version and the numbers. Leave out your employer, your customers, hostnames and credentials.

The seven parts we expect

01 · Impact

What broke, for whom

User-facing symptom, blast radius and severity: which service, which region, how long, and the SLO or error budget it consumed.

02 · Detection

How you knew

The alert, dashboard or customer report that fired first, and the gap between onset and detection.

03 · Timeline

Minute by minute, in UTC

Onset, detection, escalation, mitigation and recovery, with the decision taken at each step.

04 · Evidence

Measured, not guessed

The system views, wait events, logs and plans that located the fault. Quote the metric and the value.

05 · Root cause

Mechanism, not blame

The engine behaviour that produced the symptom, and the workload, configuration or topology change that set it off.

06 · Mitigation and rollback

What you changed, and how you would undo it

Every change with its blast radius, whether it needed a restart or failover, and the tested path back.

07 · Corrective actions

So it does not recur

Alerts, runbooks, capacity changes and design decisions, each with an owner, and what you would not repeat.

Where the evidence lives, engine by engine

Stewards expect evidence from the instrumentation each engine actually exposes. These are the surfaces a strong account cites.

Engine or serviceWhere the evidence livesA strong account names
PostgreSQLpg_stat_activity, pg_stat_statements, pg_locks, pg_stat_replication, EXPLAIN (ANALYZE, BUFFERS)Wait events, xmin horizon and transaction ID age, replication slot retention, plan changes
MySQLperformance_schema, the sys schema, SHOW ENGINE INNODB STATUS, replication_applier_status_by_workerHistory list length, redo and flushing pressure, lock waits, applier parallelism, GTID position
MariaDBperformance_schema, information_schema, Galera wsrep_% status, MaxScale diagnosticsFlow control, certification failures, cluster size changes, router decisions
SQL Serversys.dm_exec_requests, sys.dm_os_wait_stats, Query Store, sys.dm_hadr_database_replica_statesTop waits, plan regressions and forced plans, AG redo and log send queues
OracleV$SESSION, V$ACTIVE_SESSION_HISTORY, AWR and ADDM where Diagnostics Pack is licensed, otherwise Statspack; V$DATAGUARD_STATSTop wait classes, SQL ID and plan hash changes, apply and transport lag
Db2MON_GET_* table functions, db2pd, db2pd -hadrLock waits and escalations, bufferpool hit ratios, HADR log gap and state
MongoDBserverStatus, currentOp, the profiler, rs.status(), explain("executionStats")WiredTiger cache pressure, replication lag, docs examined versus returned, chunk balancing
ClickHousesystem.query_log, system.parts, system.merges, system.replication_queue, EXPLAIN PIPELINEParts per partition, merge backlog, Keeper and replication queue health, rows and bytes read per query
RedisINFO, SLOWLOG, LATENCY DOCTOR, MEMORY STATSEvictions, fragmentation ratio, fork and persistence latency, hot keys, replica offset
Cloud DBaaSRDS and Aurora Performance Insights; Azure SQL Query Store and Intelligent Insights; Cloud SQL and AlloyDB Query Insights; Redshift system views; BigQuery INFORMATION_SCHEMA.JOBS; Snowflake ACCOUNT_USAGE; Databricks system tablesWhat the provider controlled versus what you did: maintenance events, storage autoscaling, failover timing, quota and credit limits

The shape, engine by engine

Eleven illustrative accounts across relational, distributed, NoSQL, in-memory, real-time analytics and cloud warehouse platforms. Each follows the same seven parts. Open any of them to see the level of evidence a steward expects. None describes a real company; all are written to be technically accurate.

PostgreSQL Guild · PostgreSQL 15Transaction ID wraparound warning caused by an abandoned replication slot
Impact
The primary logged wraparound warnings for its largest database and disk usage on the WAL volume grew by roughly 1.2 TB over three weeks. No outage, but the database was on course to stop accepting writes to protect itself.
Detection
A disk-usage alert on the WAL volume, followed two days later by a transaction ID age alert at 1.2 billion. The disk alert was treated as routine growth for too long.
Timeline (UTC)
Day 1 09:10 disk alert · Day 3 04:30 XID age alert · 05:15 on-call finds an inactive slot · 06:40 consumer confirmed decommissioned · 07:05 slot dropped · Day 3 to 4 targeted freeze vacuums complete.
Evidence
pg_replication_slots showed a logical slot with active = false holding back catalog_xmin. age(datfrozenxid) was rising daily, pg_stat_progress_vacuum showed autovacuum completing without advancing relfrozenxid, and pg_wal_lsn_diff put retained WAL above 1 TB.
Root cause
A CDC connector was retired without dropping its logical replication slot. The slot pinned both WAL and the catalog xmin horizon, so vacuum could not freeze old tuples.
Mitigation and rollback
Confirmed with the owning team that the consumer was gone, then dropped the slot behind a written confirmation gate. Ran manual VACUUM (FREEZE) on the oldest tables during low traffic. Rollback for the drop is a re-created slot plus a full resync of the consumer, documented before the change.
Corrective actions
Alerts on inactive slots and retained WAL per slot; max_slot_wal_keep_size set as a safety cap; slot removal added to the decommissioning checklist for every CDC consumer.
Illustrative, not a record. Real accounts name the engine and version and never the employer, customer, hostnames or credentials.
MySQL and MariaDB Guild · MySQL 8.0Replica lag during month-end batch
Impact
Read replicas serving reporting fell up to 15 minutes behind the source every month-end night; balance reports were stale for roughly two hours. No data loss; the source was unaffected.
Detection
A lag alert at 300 seconds fired about 40 minutes after onset, because lag only crossed the threshold once the largest batch transactions arrived.
Timeline (UTC)
21:00 batch starts · 21:40 lag alert · 21:55 on-call confirms the pattern from previous months · 22:20 batch paused · 23:10 replicas caught up.
Evidence
Seconds_Behind_Source rose only inside the batch window. performance_schema.replication_applier_status_by_worker showed one applier worker busy while seven sat idle. The source binary log showed single transactions of several hundred thousand rows.
Root cause
Very large transactions touching overlapping rows serialised the replica’s apply, so multi-threaded replication could not parallelise the batch.
Mitigation and rollback
Split the batch into transactions of about 10,000 rows and reviewed the source’s dependency-tracking settings for parallel apply. Every change was dynamic and reversible, and was rehearsed on a staging replica first.
Corrective actions
An alert on applier worker utilisation, not only on lag; a review gate for any batch job above a set rows-per-transaction limit; and a runbook entry for pausing the batch safely.
Illustrative, not a record. Real accounts name the engine and version and never the employer, customer, hostnames or credentials.
MySQL and MariaDB Guild · MariaDB 10.11 with GaleraCluster-wide write stalls from flow control
Impact
Writes across a three-node Galera cluster stalled for several seconds at a time during a nightly data load, and application connection pools exhausted.
Detection
Application timeout alerts and a spike in pool wait time; no database alert fired because each node remained healthy.
Timeline (UTC)
01:00 load begins · 01:12 first timeouts · 01:25 on-call sees flow control paused · 01:40 load moved off the cluster’s busiest window · 02:05 stable.
Evidence
wsrep_flow_control_paused rose sharply and wsrep_local_recv_queue_avg grew on one node only. That node’s disk latency was double the others’, and wsrep_flow_control_sent showed it was the node throttling the cluster.
Root cause
One node sat on slower storage. Its receive queue crossed the flow-control limit during heavy writes, and Galera paused replication cluster-wide to let it catch up.
Mitigation and rollback
Moved the load to smaller transactions, routed it through the node with the best storage via MaxScale, and scheduled storage replacement for the slow node. Routing and batching changes were reversible within minutes.
Corrective actions
Alerts on flow control paused time and per-node receive queue; uniform storage classes across Galera members as a design rule; load jobs capped by transaction size.
Illustrative, not a record. Real accounts name the engine and version and never the employer, customer, hostnames or credentials.
SQL Server Guild · SQL Server 2019CPU saturation after a plan regression
Impact
CPU on the primary replica of an Availability Group sat above 95 percent for 50 minutes during business hours; order searches timed out for a share of users.
Detection
A CPU alert at 90 percent sustained for ten minutes, and application latency alerts shortly after.
Timeline (UTC)
05:30 statistics update job completes · 05:42 CPU climbs · 05:52 alert · 06:05 Query Store identifies the regressed query · 06:20 previous plan forced · 06:32 CPU normal.
Evidence
Query Store’s regressed-queries view showed one query_id with a new plan using a scan and key lookups optimised for an atypical parameter value. sys.dm_os_wait_stats was dominated by SOS_SCHEDULER_YIELD, and sys.dm_exec_query_stats showed the new plan’s CPU per execution more than 20 times the old one.
Root cause
A parameter-sensitive stored procedure was recompiled after the statistics update using an unrepresentative parameter value, producing a plan that was poor for the common case.
Mitigation and rollback
Forced the last good plan with sp_query_store_force_plan. Rollback is sp_query_store_unforce_plan, tested on the secondary first.
Corrective actions
A Query Store regression alert; a code fix for the procedure’s parameter sensitivity; automatic plan correction evaluated where the edition supports it; statistics jobs moved out of business hours.
Illustrative, not a record. Real accounts name the engine and version and never the employer, customer, hostnames or credentials.
Oracle, Db2 and SAP HANA Guild · Oracle Database 19c with Data GuardPhysical standby falling behind during maintenance
Impact
Apply lag on the physical standby reached 40 minutes during a nightly index maintenance window, exceeding the 5-minute RPO the standby was meant to guarantee.
Detection
A Data Guard lag check in monitoring, which only polled every 15 minutes, so the breach was seen late.
Timeline (UTC)
22:00 maintenance starts · 22:35 lag alert · 22:50 alert log shows no standby redo log available · 23:30 maintenance ends · 00:10 lag back under one minute.
Evidence
V$DATAGUARD_STATS showed apply lag climbing with transport lag near zero. V$STANDBY_LOG listed fewer standby redo log groups than online redo log groups, and the standby alert log recorded that no standby redo log was available, so redo was being archived before it could be applied in real time.
Root cause
The standby redo logs were sized and counted for the original workload. Under heavy redo generation the standby fell back from real-time apply to archive-based apply.
Mitigation and rollback
Added standby redo log groups to one more than the online redo groups per thread, at the same size, on both primary and standby. Changes were online; extra groups can be dropped if needed.
Corrective actions
Apply-lag polling every minute with an RPO-based threshold; standby redo log sizing added to the capacity review; redo generation per maintenance job tracked.
Illustrative, not a record. Real accounts name the engine and version and never the employer, customer, hostnames or credentials.
Oracle, Db2 and SAP HANA Guild · Db2 11.5 LUWLock timeouts from lock escalation
Impact
Online transactions timed out for 25 minutes while a month-end adjustment job ran; the job itself completed.
Detection
Application lock-timeout errors (SQL0911N reason code 68) and a rise in failed transactions.
Timeline (UTC)
18:00 job starts · 18:06 timeouts begin · 18:15 on-call finds escalation messages · 18:31 job completes and timeouts stop.
Evidence
db2diag.log recorded ADM5502W lock escalation messages for the job’s table. MON_GET_DATABASE showed lock_escals and lock_timeouts rising together, and MON_GET_LOCKS showed a table-level lock held by the job.
Root cause
The job updated several hundred thousand rows in one unit of work. Its row locks exhausted the lock list share allowed by MAXLOCKS, and Db2 escalated to a table lock that blocked online work.
Mitigation and rollback
Changed the job to commit every 5,000 rows and set LOCKLIST and MAXLOCKS to AUTOMATIC under the self-tuning memory manager. Configuration changes were online and reversible.
Corrective actions
An alert on lock escalations; commit-interval standards for batch jobs; job scheduling moved away from peak online hours.
Illustrative, not a record. Real accounts name the engine and version and never the employer, customer, hostnames or credentials.
NoSQL Guild · MongoDB 6.0Write stalls and secondary lag from cache pressure
Impact
Write latency rose from single-digit milliseconds to over a second for 30 minutes, and secondaries fell several minutes behind, delaying majority-acknowledged writes.
Detection
Application latency alerts first, then a replication lag alert.
Timeline (UTC)
02:00 bulk upsert job starts · 02:08 latency climbs · 02:20 on-call sees dirty cache above threshold · 02:35 job paused · 02:50 recovered.
Evidence
serverStatus showed WiredTiger tracked dirty bytes well above the eviction trigger and application threads performing eviction. The profiler showed the job’s upserts with docsExamined in the millions against nReturned of one. rs.printSecondaryReplicationInfo() confirmed lag on every secondary.
Root cause
The upsert filter field had no index, so each upsert scanned the collection, flooding the cache and forcing application threads into eviction.
Mitigation and rollback
Paused the job, built an index on the filter field during low traffic, and re-ran the job with throttling and majority write concern. Rollback is dropping the index.
Corrective actions
Index review for every new batch job; alerts on dirty cache ratio and application-thread eviction; job throttling as a default.
Illustrative, not a record. Real accounts name the engine and version and never the employer, customer, hostnames or credentials.
NoSQL Guild · Apache Cassandra 4.1Read timeouts from tombstone accumulation
Impact
Reads on a work-queue table timed out for a growing share of requests over a week, until one partition became unreadable.
Detection
Rising client read-timeout errors and tombstone warnings in system.log; no alert existed for either.
Timeline (UTC)
Week 1 warnings begin · Day 7 09:00 timeouts exceed 5 percent · 10:30 tombstone-heavy partitions identified · Day 8 interim fix · Day 21 redesign shipped.
Evidence
system.log reported reads scanning tens of thousands of tombstone cells for a handful of live rows. nodetool tablestats showed high tombstones per slice for the table, concentrated in a few partitions.
Root cause
The table was used as a queue: items were inserted and deleted in the same partitions. Deletes left tombstones that persist until gc_grace_seconds expires and compaction removes them, so reads scanned ever more dead cells.
Mitigation and rollback
Interim: lowered gc_grace_seconds only after confirming repairs completed well inside the new window, and ran targeted compaction. Long term: redesigned to time-bucketed partitions with TTL and TimeWindowCompactionStrategy. The interim setting was reversible.
Corrective actions
Alerts on tombstone warnings; a design review rule against queue patterns on Cassandra; repair cadence monitored against gc_grace_seconds.
Illustrative, not a record. Real accounts name the engine and version and never the employer, customer, hostnames or credentials.
Redis and Valkey Guild · Redis 7.2Periodic latency spikes during snapshots
Impact
p99 latency for a session store jumped from about 1 ms to 300 ms every 15 minutes, causing intermittent login failures.
Detection
Synthetic login checks failing on a regular cadence, which pointed to a scheduled cause.
Timeline (UTC)
Spikes aligned to every 15 minutes · 11:20 on-call correlates spikes with BGSAVE · 12:00 fix applied on a replica first · 13:30 primary updated.
Evidence
INFO persistence showed latest_fork_usec near 280,000 on a 25 GB dataset. LATENCY HISTORY fork matched the spikes, and the host had Transparent Huge Pages enabled.
Root cause
Forking a large process with Transparent Huge Pages enabled made the fork and subsequent copy-on-write expensive, blocking the event loop during each snapshot.
Mitigation and rollback
Disabled Transparent Huge Pages on the hosts and moved RDB snapshots to a replica. Both changes are reversible, and were applied to the replica before the primary.
Corrective actions
An alert on latest_fork_usec; host baseline checks for Transparent Huge Pages; persistence strategy documented per role.
Illustrative, not a record. Real accounts name the engine and version and never the employer, customer, hostnames or credentials.
Real-time Analytics Guild · ClickHouse 24.8Inserts rejected with too many parts
Impact
Event ingestion from Kafka was rejected for 35 minutes; consumers retried and lag reached several million messages, delaying dashboards.
Detection
Consumer lag alerts and TOO_MANY_PARTS exceptions in the ingestion service logs.
Timeline (UTC)
14:00 new tenant onboarded · 14:20 inserts rejected · 14:30 parts count confirmed · 14:45 batching changed · 15:30 merges caught up, lag cleared by 16:10.
Evidence
system.parts showed thousands of active parts in single partitions. system.merges showed a growing backlog, and system.part_log confirmed one new part per insert at a high insert rate.
Root cause
Consumers inserted very small batches many times per second into a table partitioned by hour and tenant. Part creation outpaced merges until the parts_to_throw_insert threshold rejected inserts.
Mitigation and rollback
Increased batch size and enabled asynchronous inserts for the consumers, and moved the partition key to daily for new data. Batching changes reverted in minutes; the partition change was staged through a new table and swap.
Corrective actions
An alert on maximum active parts per partition; a partition-key review for every new table; ingestion batching standards.
Illustrative, not a record. Real accounts name the engine and version and never the employer, customer, hostnames or credentials.
Cloud Warehouse and Lakehouse Guild · SnowflakeCredit spend doubling with queued BI queries
Impact
Monthly credit consumption roughly doubled over a quarter, while BI dashboards still showed queuing at peak hours.
Detection
A finance review of the monthly invoice, not monitoring; there was no budget alert.
Timeline (UTC)
Month 1 to 3 gradual growth · Month 3 review flags the invoice · two weeks of analysis · changes rolled out over one sprint.
Evidence
ACCOUNT_USAGE.WAREHOUSE_METERING_HISTORY showed one large warehouse running nearly around the clock. QUERY_HISTORY showed BI queries with high queued_overload_time behind ELT jobs that spilled heavily to remote storage.
Root cause
BI and ELT shared one warehouse with a long auto-suspend setting. ELT spilling inflated runtimes, the warehouse rarely suspended, and BI queued behind it.
Mitigation and rollback
Split BI and ELT onto separate warehouses sized for each, set short auto-suspend, rewrote the worst-spilling ELT steps, and added resource monitors. All settings are reversible per warehouse.
Corrective actions
Resource monitors with alerts at budget thresholds; weekly review of spill and queued time per warehouse; warehouse ownership per team.
Illustrative, not a record. Real accounts name the engine and version and never the employer, customer, hostnames or credentials.

About 300 words is enough; the form caps the account at 500. Write for a peer who runs the same engine: they will notice what you measured, what you ruled out, and how you would roll back.

How review works

Step 1 · within two working days

A person replies

The secretariat confirms receipt, checks eligibility and routes your application. No autoresponders.

Step 2 · within two weeks

A steward reads it

Guild applications go to a steward of the guild you chose first. Circle requests go to the members’ council.

Step 3 · if needed

A 30-minute conversation

If the steward wants more context, they walk through your account with you. Engine to engine, no trick questions, nothing recorded.

Step 4 · on admission

Welcome and introductions

A profile call, three warm introductions in your first month, and your first guild session or Circle working session.

If we cannot admit you yet, we say why and when to apply again. Nobody is turned down without a reason.

What happens to your application data

What you give usWhy we need itHow long we keep it
Name, work email, LinkedInTo reply to you and confirm who you areUntil your request is resolved; while you are a member if admitted
Role, years, sector, center stage, cityTo route you to the right guild, cohort and chapterAs above
Engines, versions, deployment, scale, HA/DRTo let a steward assess your practiceAs above; never shared outside stewards and the secretariat
Production account or decisionEvidence for review, and a possible first sessionAs above; never published without your consent
Nominee detailsTo contact the person you nominatedUntil the nominee accepts or declines

Nothing you send is added to a marketing list or passed to the commercial teams of MinervaDB or ChistaDATA. Full detail in the privacy policy.

Nominations

Choose “Nominate a peer” in the form. Tell us what the nominee runs and a problem you have seen them solve. We contact them directly, explain the club, and tell them who nominated them only if you say yes. The council treats a nomination from someone who could have taken the seat as the strongest recommendation there is.

The form

About ten minutes. Fields marked * are required. Sections appear once you choose your path in section 2. Near the end, an optional Special Interest Groups section lets you pick the SIGs you want to work in.

1 · About you

Seats and memberships are personal. Tell us about you, not your employer. Nothing here is shared outside the secretariat and the council.
Optional. Helps the steward or council place your experience quickly.
Center stage

2 · Your request

Choose one. The form shows only the questions that apply to your path.
I am writing to

Special Interest Groups (optional)

SIGs are the smaller working groups inside the guilds, one per platform plus one for cloud DBaaS, where members write the runbooks, drills and reference designs. Choose any that match what you run in production. Nothing here affects admission, and you can change it later. See the ten SIGs.
Which SIGs interest you?
Select all that apply.

Final details

Questions, context or constraints. If you are asking the secretariat a question, write it here.
Eligibility
Charter and privacy
Member directory
Optional. The directory is visible only to admitted members and is never exported.

Questions

Is there a fee?

No, in either tier, now or later. The price of membership is one contribution a year.

We run everything on managed services. Is that enough?

Yes. RDS, Aurora, Azure SQL, Cloud SQL, AlloyDB, BigQuery, Snowflake and Databricks all count. Managed services sit in the guild of the engine underneath them.

Can I join a guild for an engine I am migrating to?

Yes. Apply on the engines you run today and add the target guild. Migration experience is exactly what the Migration practice wants.

My best incident involves confidential details.

Describe the mechanism, not the company. Engine, version, metrics and the fix are enough; never include hostnames, credentials or customer data.

Do I need to be on call?

No. On-call position helps us place you, but architects and engineers who own design rather than pages are welcome.

Can I apply to more than one guild?

Yes. Most members join two or three. Tick each one in the form.

I work at the parent company, not the GCC.

You are eligible if you sit in the parent’s data organisation and work with a center in India.

I am in sales or pre-sales.

Then you are not eligible, at any company, including MinervaDB and ChistaDATA. You are welcome at the open conferences.

Anything else: choose “Ask the secretariat a question” in the form, or write to shiv@minervadb.com.