I always increase the delegated amount, but less and less rewards are transferred. Shouldn’t it be the other way around?

This is due to the fact that the stakes of the largest masternodes are growing in the network; accordingly, the share of rewards decreases for all others. In the test network, hikes in rewards are possible because of the unlimited volumes of test coins. The mechanism for determining rewards for delegation and an example of calculation can be found in this section of the documentation https://docs.minter.network/#section/Delegator-FAQ