Compared to a lot of other languages around, they really aren't. They are, however, very different from more commonly studied languages, like Germanic and Romantic languages, and that presents challenges for many learners.
Some main ways that Celtic languages seem difficult for learners, in my many years of experience with Celtic languages:
-- Celtic languages are VSO (verb-subject-object) languages. Many of world's most studied languages, like English, Mandarin, French, Spanish and Japanese, are either SVO or SOV. Surprisingly, VSO word order is not that hard once you get the hang of it, but a lot of people are scared by it because it's different.
-- They contain phonemes not found in many other languages, and in some cases, Celtic language have more phonemes to learn than the learner's native language. Learners often find these unfamiliar sounds difficult to master. This is one of the more genuinely difficult aspects of learning a Celtic language.
-- Orthography for the Celtic languages is very unique. Since Celtic languages have their own sounds, they need their own orthography that reflects that. And none of the 6 modern Celtic languages use the same orthography. The closest are Irish and Scottish Gaelic, which are both based in the same older orthography for Old and Middle Irish, but today are distinct from each other.
-- There are grammatical concepts that are pretty much unique to the Celtic language. For example, they don't have an infinitive form of the verb. Instead they use the verbal noun, or verb-noun, which can function similarly to an infinitive in Romantic languages, but are not limited to that. As the name suggests, they can be used as a verb or noun, depending on the context, and the way they are used is a very new way of thinking in a language from what most learners already know.
-- Variations among dialects and between colloquial and more formal usage can be very confusing and frustrating for learners to navigate. This is not unique to the Celtic languages, as most language learners encounter this in nearly any language they might study. But within the small world of the Celtic languages, this can get pretty complicated and daunting. And as is the case with nearly any minority language in the world, things get political very quickly. This is why I tend to avoid Irish, why Breton can get so very, very aggravating for me and why I came close to giving up Scottish Gaelic more times than I can recall. You won't be able to avoid politics with any language, naturally, but with minority languages, the room is already small to begin with, so when someone shows up with really big opinions about what they think is correct and proper and right, it can feel a bit...suffocating. And if you're a learner just trying to get a handle of the basics, that can be really off-putting.