Just my two cents... I've found French to be a big pain in the ass, not because of its grammar or pronunciation, but because of how its vocabulary works. There are just so many different things that are said differently than English, and that's what takes forever and a half to learn, and what makes the difference between natural French and foreigner French.
An example I learned today:
"I forget the last time we spoke." is NOT "J'oublie la dernière fois où nous avons parlé." but more like "J'ai oublié à quand remontait la dernière fois où nous avons parlé."
It's just a neverending string of vocabulary and expressions to learn. I guess that's how every foreign language is, though... Some are easier to get up to speed in, but they all take a lifetime to get truly good at.
On the other hand, I've been finding Finnish to be very difficult to make a basic foundation in, because to even make a basic sentence, you need to know about consonant gradation, verb types, and how to form cases, and all of that takes a rather long time to get set up properly.
Swedish is proving very difficult because of its pronunciation. The grammar is mercifully similar to English's in many ways, but when I stumble over my own tongue to actually say the words, it doesn't help much