Yip, tones ARE difficult if you are not yet a speaker of a tonal language!
That does not mean they are insurmountable. After several years of training I've found I'm becoming better at it (not just pronouncing them more or less right but actually hearing them). You might well be faster or slower, depending on how much effort you put.
If you are not a speaker of a language which a large sinitic component yet (Japanese, Vietnamese), you'll find the vocabulary difficult, too, as it has no links whatsoever with any indoeuropean language (that's pretty obvious, but it still slowed me down terribly until I had some basic vocabulary in place).
Still a great language to learn if you're in for a serious commitment.