arXiv:2601.16812v2 Announce Type: replace-cross Abstract: In many learning tasks, certain requirements on the processing of individual data samples can arguably be formalized, in an intelligible way, as strict constraints of the underlying optimization problem, rather than by means of arbitrary penalties that often require heavy trial-and-error tuning. In this paper we show that, in these scenarios, learning can be carried out exploiting a sequential penalty method that allows to explicitly deal with constraints. For the proposed algorithm we show that, under classical assumptions and in the considered learning scenario, penalty subproblems can provably be solved in finite time with suitable guarantees of approximate stationarity in expectation; we also show that, when the latter condition is guaranteed for the subproblems, the overall sequential scheme possesses almost sure convergence properties to KKT-stationary points of (general) constrained problems. The results of experiments on sandbox and real-world image processing tasks show that the method is indeed viable to be used in practical deep learning scenarios.