Researchers from Dharma-AI published an analysis on November 18, 2026, comparing analytical FLOP-based duration estimates with learned predictors for end-to-end supervised fine-tuning (SFT) jobs. Using 272 production runs on NVIDIA H200 GPUs (about 7,382 GPU-hours), the team shows that a CatBoost regressor built from job configuration and metadata predicts wall-clock time far more accurately than a compute-only FLOP baseline.
Data, baseline and prediction task
The dataset covers SFT runs up to 15 billion parameters, including both LoRA and full fine-tuning, executed on 1, 2 or 4 H200 GPUs across multi-tenant clusters. Measured durations ranged from 1.6 to 172.4 hours (mean 23.43 h, median 15.29 h). The FLOP baseline uses the common formula T_theoretical = (6 · P · D) / (N_GPU · FLOPS_peak · MFU), with MFU swept from 0.25 to 0.65. Even with MFU tuned, the FLOP baseline left large errors: for MFU=30% the baseline achieved R²=0.31 and MAE=12.41 h.
Model, features and results
The team framed end-to-end duration as a supervised regression problem using only information available before dispatch: hyperparameters, dataset sizes, model identifiers, and GPU allocation. Derived features exposed overheads such as num_eval_data, num_saves, num_params_saved and per-GPU token volumes. The FLOP estimate with MFU=0.45 was included as a model feature.
Across stratified 5-fold cross-validation, CatBoost using 24 features reached R²=0.98, RMSE=3.52 h and MAE=1.71 h (median absolute error 0.51 h). XGBoost showed similar performance (R²≈0.97, MAE≈1.95 h). Linear models and analytical alternatives performed worse: Ridge MAE was 7.23 h, and an extended implementation of the analytical Calculon model yielded a median covering only ~11% of observed wall-clock time.
Feature-importance and SHAP analyses placed evaluation and checkpoint variables among the top predictors: num_eval_data ranked first, followed by num_trainable_params, batch_eval_size_percent, num_all_params and num_gpus*batch_train_percent, with num_params_saved also relevant. The authors note that num_gpus partly proxies for job scale in their data.
Predictor accuracy remained strong for runs up to 60 hours (median absolute error ≈1 h); errors widened in the sparse tail above 60 hours. Limitations include focus on SFT only, a single GPU type (H200), and few very long runs. The full paper, titled “Predicting End-to-End Fine-Tuning Time for Generative Language Models,” will be posted on arXiv soon; next work items include hardware-specific descriptors, validation on other GPU types, transfer learning for sparse configurations and more long-run data.
Original source: Hugging Face Blog