RLHF data is mostly generated by AI

In reinforcement learning from human feedback, humans are used only to train the reward function; once trained, the reward function's interaction with the model is automatic, so most of the data generated during reinforcement learning is created by the AI rather than by humans.