The Spirit Realm

Both factions are one-way trips to Ally. There's no mob I know of that lowers either faction for killing it, or tasks/missions/quests/etc. Further, the only mobs to kill that raise these factions are the instanced targets that require some raid that is already 85% Ally. The factions are entirely task-driven.

There are no negative effects for either faction going up.

The other factions that matter to 75+ (especially "+", more accurately "++") raiding are Crusade of Scale, Wraithguard, and Wraithguard Leadership.

These two factions don't conflict with each other. From what we can tell, you need ally in both factions in order to zone into the final Ashengate raid instances. Doing every quest (10 for Infiltraitors, 11 for Scholars, plus misc flagging events) is more than enough to get you Ally in the applicable faction.

As far as other factions to worry about, as Brohg said, there's 'Wraithguard' and 'Wraithguard Leaders' factions on the Frostcrypt side. I myself don't know as much about here as the AG side.

As for grouping in certain areas of certain zones, this really only comes into play in Valdeholm. The giants between the Zin and the gate to the city proper, are all on Wraithguard faction. Leave them alone. It only takes a few kills to lower your faction to abysmal levels, requiring you to pretty much live in the deeper parts of Frostcrypt till you get high enough for them again. Once you go through the gate of the city, all those giants are on 'Valdeholm Citizens' faction, which even if your faction with them can't get any better, they're all still KoS. Those are just fine to slaughter to your heart's content.

Faction hits for kills used to be really crappy untill helloween then they raised faction hits per kill in deeper FC drastically. I guess neg faction hits for killing Wraithguard outside were raised too. I killed like 10+ wraithguard without any visible faction drop before that change.