So I found some more Jersey fabric I had stashed away from when I first got my sewing machine and decided to put it to good use by making another Lydia Top. This one's got long sleeves like the pattern intended and will come in handy seeing as it's heading into winter now.
I followed the pattern exactly as before but again hit a wall trying to get the armholes to make any kind of sense when pinning and sewing them. I decided the pattern wouldn't get the better of me and cut off about 2 or 3 centimetres of extra fabric around the shoulder of the sleeve and voila! It sewed up perfectly first time. No puffiness, no swearing, just plain old flat shoulder seams. Hooray!