Defender diversity means different variants, we lose points for repeating them. Example: If I use two Harlequin Bellas, I'll lose 100 points, and so on. It have nothing to do with "not using a fighter of the game". I don't use any Eliza and Painwheel but still get the 2000 points.Defender diversity means you can use any fighter you want, but if you exclude any fighter of the roaster, you will lose points.
I don´t use any Beowulf, so I got a 100 points penalty, if I don´t use any cerebella´s at all, it would be -200 points, and so on. So if you want to avoid this kind of problem, use all avaible characters in your rift defense map, or get a better score with block buster finish, no companion deaths, etc...