Defender diversity means you can use any fighter you want, but if you exclude any fighter of the roaster, you will lose points.
I don´t use any Beowulf, so I got a 100 points penalty, if I don´t use any cerebella´s at all, it would be -200 points, and so on. So if you want to avoid this kind of problem, use all avaible characters in your rift defense map, or get a better score with block buster finish, no companion deaths, etc...