A Flight Academy’s Double-Charge Weekend and the ServerlessOps Fix

A Flight Academy’s Double-Charge Weekend and the ServerlessOps Fix

Picture a small flight academy somewhere in India that sells block bookings: ten hours in a trainer, a briefing slot, and a ground-school seat, all paid through one online form. The team behind its website is three people and a part-time contractor. They keep the booking site on event-driven cloud functions because nobody wants to babysit a server at midnight. The tale is invented, stitched together from weekends that ServerlessOps practitioners describe, but every step in it is the sort of thing that really does happen when function-based systems meet an unreliable payment provider.

It begins on a Friday evening, the night before a cohort of trainee pilots is due to confirm their slots. Students sit at home, aspirants who earlier spent weeks on material like a plan for preparing for the aptitude exams, and finally click the pay button. Some of them click twice because the page seemed to hang. That small human habit is the first character in our story.

The Saturday morning surprise

By Saturday, the front desk had eleven emails from students who had been billed twice for the same block. Nobody had touched the code since Tuesday. The team lead, a former ground instructor turned developer, opened the dashboards and found nothing red. Error counts were flat. The functions had done exactly what they were told, and then done it again.

The contractor traced it in an hour. The payment provider had been slow, so the function that charges the card timed out on its side while the charge itself had actually gone through. The queue in front of the function saw a failure, waited, and delivered the same message again. A second function run, a second charge. Nothing was broken in the way an alarm understands broken. The platform simply promises at-least-once delivery, which is a fair promise and a trap for anyone who read it as exactly-once.

That is the very first lesson in the discipline. The Ops Atlas entry on ServerlessOps lists trusting platform exactly-once as the opening anti-pattern, and the story above is what it looks like in a flight-school budget.

Why the cockpit analogy helps

Instructors in a cockpit already live by a rule that fits this problem. When a student repeats a checklist item because the callout was unclear, the aircraft does not break; a second pass over the same switch simply does no further harm, provided the switch is already in the commanded position. Checklists are written to be safe when repeated. A charge function is a switch that was never designed that way.

The fix the team eventually adopted is the software version of making the checklist repeat-safe. Before charging, the function looks up a record keyed on the booking itself: this student, this block, this date. If a record exists, it returns the earlier answer and does nothing further. If it does not, it writes the record, charges, and stores the outcome. The key must be something the business owns, not the message identifier the queue assigns, because a redelivery or a manual redrive arrives with a new identifier while the booking stays the same.

Testing the idea before trusting it

The contractor wanted proof, not optimism. She built a small simulation using the pattern the Atlas describes: 10,000 orders where 2% were repeated by impatient users, three percent of charges timing out after the money had already moved. Three versions of the code ran against the same input.

  • Without any protection, 504 customers were billed twice.
  • A version that only recorded the message identifier still double-charged 295 times.
  • The version that stored a record on the business key, and passed that key to the payment provider too, double-charged zero times.

These numbers come from the Atlas’s own simulation, so they are a teaching device rather than a measurement of any real shop. What mattered to the team was the ordering. Recording something helped. Recording the right thing finished the job. The academy owner, who had expected a long explanation, understood it in the time it took to read three lines.

The second hole: a dead-letter box nobody owned

While cleaning up, the contractor found a second, quieter problem. Messages that exhausted their retries had been landing in a dead-letter queue for weeks. Nobody was watching it. Twelve of those were enrolment confirmations that never reached students. In flying terms, it was a lost-and-found shelf with no attendant.

The remedy followed the Atlas’s guidance for any asynchronous path: name a keeper for the dead-letter destination, wire an alarm to it, and let it hold messages for longer than the queue feeding it. The page offers an example of fourteen days against four on the source. The reasoning is plain once you picture a long outage. If the original queue discards messages after four days and the failure box also forgets after four, a long weekend of trouble erases the evidence you need. With fourteen days, the team can come back on Monday, read what failed, and replay it in an orderly way.

They named the front-desk lead as owner, because she is the person who actually knows what an unconfirmed booking means to a student, and routed the alarm to the same channel the ground instructors already read each morning.

What they refused to do

The easy overreaction would have been to add retries at every layer: the browser, the gateway, the function, the queue consumer. The Atlas warns against this with a vivid piece of arithmetic attributed to Amazon engineering advice. Stack five layers that each retry three times and one failing call multiplies into 243 attempts against a struggling dependency. An overloaded payment provider would have received a storm instead of a trickle. The team chose to retry at one layer only and to let the rest fail fast.

They also capped concurrency for the charging function. A function that scales without limit can take a downstream system down with it. The Atlas suggests holding peak concurrency below seventy percent of the cap, which gave the academy room to absorb a surge on enrolment day without starving its other functions.

A calmer Monday

On Monday, the team refunded the eleven duplicate charges by hand and wrote to each student with a short, plain apology. Then they did something unglamorous: they made a short list of the five functions that cause real-world side effects, which were charging, sending the confirmation, issuing the ground-school seat, writing to the logbook system, and emailing the instructor roster, and put the repeat-safe pattern on each. The Atlas calls this the sensible middle stretch of a ninety-day plan, and the team simply followed it without ceremony.

Trainees who care about this sort of reliability tend to be the same people who read widely before committing to a career in the air. If you are one of them, an overview of entrance routes into aviation courses is a good companion for your planning, and the habits in this story carry over: write the check down, make it safe to repeat, and know who owns the result.

Lessons worth taking back to your own desk

Notice that the story never needed a new product. The academy did not buy a monitoring suite or hire a consultancy. Three habits closed the gap, and each one translates directly to the cockpit.

  1. Treat every repeat as normal. In aviation, a go-around is routine; in functions, a redelivery is routine. Design for it.
  2. Key your records on the thing the business cares about. The student and the block are stable. A message number is not.
  3. Give every safety net a named keeper. A dead-letter queue with no owner is an unattended lost-property shelf.

One more point from the entry: the label is historical. ServerlessOps was a niche practitioner term from roughly 2016 to 2020, and no standard body or certificate stands behind it, yet the entry observes that the underlying practice is now mainstream. That is a comfortable fact for a small academy: you are not missing a credential, you are missing a handful of habits.

One last check on the numbers that matter

After the fixes, the team agreed on a few measures to glance at weekly. The Atlas names several, and they chose three that a front-desk person could actually read. First, the share of bookings completed successfully from the customer’s point of view, not merely the share of function runs that returned without error. Second, the rate of duplicate side effects, which they wanted to see sitting at zero. Third, how old the oldest item in the failure box is, because age is the quickest tell that nobody is looking.

If you are building a small booking site for a training school, you can try the same experiment this week. Send the same booking twice, delay the payment response, and see what your system does. If the answer is anything other than one charge and one confirmation, you have found your Saturday morning before it finds you.

Comments

No comments yet. Why don’t you start the discussion?

Leave a Reply

Your email address will not be published. Required fields are marked *