When it comes to validating reduced order models (ROMs), there are many potential means and methods that need a clear consensus approach, with results that are often dependent on the choice of both the training and the validation sets. In this study, we investigated a range of error metrics and methods for validating ROMs. The purpose was to identify the most useful metric for gauging the accuracy and robustness of a reduced order model. Specifically, we focused on a ROM based on Proper Orthogonal Decomposition (POD) with interpolation. We broke up the ROM error into two independent components: the POD representation error of the physical field variables and the interpolation error introduced by response surface methodologies (RSM), used to predict the coefficients of the POD expansion for out-of-sample configurations. To assess the contribution of each individual component, as well as the ROM performance, we selected as a verification case the prediction of the transonic flow past a RAE airfoil subject to shape deformation. In particular, we looked at the airfoil pressure and skin friction distributions and the corresponding aerodynamic coefficients. We examined potential validation metrics and identified which best corresponds to accurate predictions for a gold standard of runs with the simulation model, considering both distributed and integral quantities of interest. Given a randomly generated validation set, we quantified the aggregated error and its components as a function of the distance of the validation set from a training set of variable size to highlight any potential bias in its construction. In particular, we considered not only the notion of distance in the parameter space, but also in the solution space and in terms of actual distance among the airfoil geometries, to identify any misleading metrics. In addition, looking at the error metrics, we studied and described the tuning of some ROM hyperparameters, such as the number of POD leading modes and the RSM hyperparameters.