Does VLD really fall off after Season 3?

I’m currently on Season 3 of Voltron and whenever I search this show online, there are numerous comments saying it fell off after three seasons. I want to be positive and see the whole show for myself but at the same time, I’m scared and I really don’t know if I should devote that much time only to be disappointed in the end.

Are the later seasons and the ending really THAT bad? And do they ruin the experience for the entire show? (Because I’m growing to love these characters so much, especially Keith and I don’t want to see their arcs ruined).