Ancient magic spells are Black Mage spells that deal large amounts of damage at the cost of large amounts of MP and long cast times. The first tier of spells are rarely used because of these drawbacks and should not be used for normal nuking.
List of Ancient Magic Spells
Soloing with Ancient Magic
One technique Black Mages can use to kill things unaided is by taking advantage of the long cast times of Ancient Magic. While this is usually a drawback, this can benefit the mage when used with Sleep II. The Black Mage just has to cast Sleep II on the foe, and then cast an Ancient Magic. When the spell completes casting, they sleep it again. If they have MP to continue, they repeat this process. If not, they can somewhat safely rest to get some MP back. Next, they rest until they either each full or the monster wakes up.
The long cast times of Ancient Magic spells allow enough time for Sleep II to be recast. If this method were attempted with normal nukes, it would probably not be successful.
Ancient Magic in Parties
While the use of Ancient Magic in traditional PTs is limited, it used to be popular to burst Freeze on distortion skillchains. This practice has fallen out of favor due to high MP cost and the fact that if Freeze didn't kill the mob, the mob would usually kill the BLM.
AM is occasionally used in "manaburn" PTs, PTs consisting of primarily BLM, with a refresher or two thrown in to keep the PT killing fast. Typically, the refresher (a RDM or BRD) will pull the monster to camp, where it will be slept. Then, the BLMs will cast an AM spell (usually freeze) in unison, usually using the ingame clock, or a macro, to keep the start time close. Typically, four AM will kill any low IT/high VT monster instantly.