OpenAI published an account from Nathan Bronson, Member of Technical Staff, describing an investigation into recurring Rockset crashes that disrupted ChatGPT’s data search infrastructure. The symptoms included C++ functions appearing to return to invalid addresses, sometimes with a NULL saved return address or an %rsp register misaligned by 8 bytes; crashes always occurred on return.
From single-case debugging to population analysis
Initial, detailed inspection of individual core dumps and register state failed to produce a single consistent hypothesis. Rockset’s execution layer is written in C++ and uses folly’s fatal signal handler to log stack traces; core dumps were uploaded to Azure blob storage. The team first followed a “doctor” approach—deep analysis of particular cores—but then shifted to an “epidemiologist” approach and built an automated pipeline to analyze every production core dump from the prior year.
The pipeline, whose early implementation was assisted by a script generated with the help of ChatGPT, extracted registers and labeled crashes as return-to-null, misaligned-stack, or other. That population-level data revealed two distinct clusters that had previously been conflated.
Two distinct causes and concrete mitigations
One cluster of misaligned-%rsp crashes traced back to a single physical Azure host exhibiting silent hardware corruption. Those nodes were denylisted and operational changes were introduced: the fatal signal handler was enhanced to include register state so similar events can be detected from logs, the control plane was adjusted so VMs are reused more often (easing bad-node detection), and runbooks were updated.
The other cluster—return-to-null crashes—were linked to exception unwinding. OpenAI found that GNU libunwind synthesizes a ucontext_t on the stack and then calls an internal assembly routine, _Ux86_64_setcontext, to apply the unwind destination state. Because the synthesized ucontext_t can live in a frame that is being unwound, a frequent SIGUSR2 delivery (scheduled via timer_create with CLOCK_THREAD_CPUTIME_ID for a coarse_thread_cputime_clock) could clobber that struct before _Ux86_64_setcontext finished, producing a NULL or otherwise invalid instruction pointer.
OpenAI reported that separating the crash populations enabled targeted remediation and informed further fixes and monitoring to prevent recurrence.
Original source: OpenAI News