Stifter's Sengoidelc 6.8.1.2:
o-stem words of one syllable that have é as their vowel in the nom. sing. (!) are more complicated. [...] If a syllable containing u follows or in case of u-inflection (pre. sg. and acc. pl.), é changes to the diphthong /ēu̯/, spelled éu, éo
Aineóil is gen. sg. aineól which is just eól plus the privative prefix. This is apparently from PIE *i-tlo-m.
I'm not familiar with the historical phonetics behind each case, but it seems kind of intuitive.
edit: I probably misread your post, nevermind. You might want to look at David Greene's "The Diphthongs of Old Irish.". The journal it's in is participating in JSTOR's Register and Read program.