Horror is incredibly subjective. So many elements have to come together in order for it to run smoothly. I think the biggest challenges are oldest:
Can the director establish and maintain tension cinematically?
Can the director/screenwriter/actors create a story with characters that we can actually in emotionally?
Sure overuse of jump scares are predictable and obnoxious, but I'm willing to bet that The Conjuring had just as many "jump scares" as Annabelle. In fact I'd bet money somewhere inside BlumHouse theres a template recommending how many often there needs to be a "jump scare". The reason The Conjuring was better was because it had a better story, better characters, and was more satisfying emotionally. Horror movies need to be built around strong characters, but usually we get weak or under written characters that only exist to serve the premise.