It does feel frustrating, that's for sure.
Though that doesn't mean it's exactly unfair, because that's just how streaks work.
You work your victory streaks up for more points, meaning you obviously fight stronger foes as you rack up streaks.
And if you lose, well, you just try again, but this...