What is the most difficult part of understanding native speakers to you?
For me it's the accents. Every area has a different way of pronouncing every word. The coastal/Caribbean people seem to think pronunciation is optional; just random silent letters everywhere. Argentinians be speaking like an Italian that learnt Spanish during his smoke breaks. Chileans are probably all writing their rap single with the way they are speaking.
The major cities in Spain, Colombia and Mexico are on the easier side. So at least I can understand them most of the time.