Three questions that belong together
Approximation, optimisation and generalisation describe different parts of learning. A function may be representable yet difficult to learn; a training procedure may fit the observations yet perform poorly on a new distribution.
Representation
Approximation and structure
Depth and width influence the class of functions available to a network. Useful questions concern approximation error at a fixed parameter budget, regularity of the target, and whether known symmetries can reduce unnecessary flexibility.
Algorithms
Optimisation and dynamics
The loss surface, parameterisation and update rule jointly shape training. Track sensitivity to initialisation, learning rate and batch size. Similar final training losses can conceal different trajectories and different predictive behaviour.
Evidence
Generalisation and stability
Separate error on a finite test sample from performance on the target population. Study how sample size, model complexity and regularisation interact, and state assumptions about independence and distribution shift explicitly.