下坡路

go downhill; be on the decline; 走下坡路

At that time our relations were going upwards, but since then they have been going downwards [downhill]. 那時我們的關系是走上坡路, 此后就開始走下坡路了。