What makes an indicator to be considered stationary, therefore okay to use in an RL model? For example, what makes RSI stationary but MA and Stochastic non stationary?
Stationarity in the statistical sense is a well-defined concept. In terms of technical indicators, what you are looking for is some degree of invariance of the signal's signature. This is required so that when you use a technical indicator as model feature input, your model does not have to deal with unbounded (arbitarily large or small) values. Most model will break down under such condition. In this broader sense, any technical indicator that transforms the input (prices or volumes) to a range bound output (e.g. RSI) can be considered suitable for use as a model feature. Whereas, unbounded transformation (e.g. moving average, which is just an average of underlying unbounded price and hence unbounded itself) are not suitable for use as a model feature.