Well, to do it entirely Hermetically, what you'd do first is find out a bit about the demon in question, and gather any arcane connections you can to him/it/her/(other sex).
Get yourself a nice, large, solidly constructed room with as few things to destroy in it as possible, and invent a containment ring spell like Circular Ward Against Demons except with the magic barrier facing inwards. Make sure that this spell is of many times higher strength than the demon, and test its strength before the next part.
Then, cast Opening the Intangible Tunnel at level 60 or higher. This is something I don't recommend you do without plenty of backup, and maybe a Wizard's Communion. I also strongly suggest you cast every form of warding you can on the (lead) caster of Opening the Intangible Tunnel.
Then, cast Wrench the Fallen Angel (level 40) and put it in the containment circle.
Wrench The Fallen Angel (40, Rego Vim)
R: Touch D: Mom T: Ind
Star Ruby (+3)
Allows the caster to teleport the demon they are in contact with (mystically or otherwise) to any point in the caster's sight or that the caster has an arcane connection to. An Intelligence + Finesse roll of 9+ is necessary to get the demon exactly where the caster wants them, although if you're merely trying to put the demon in a very large pit or circle the roll could be as low as 3+. Botches can do odd things like fuse the demon with a stone wall or other such intriguing effects.
Then, drop the Intangible Tunnel and Arcane Connection so fast that the demon doesn't have time to strike (hopefully).
Now, if you've done any spell manufacturing or modification for Ars Magica, you may be curious as to why I suggested using so high a level teleportation spell. If you are only a few feet away from a containment circle, why couldn't you just lower the range of the teleportation to - for instance - Near and shave three magnitudes off the last two spells (making them levels 45+ and 25)?
Well, you could do that. However, in my case, my magi are keeping their options open. With this high level a spell you can a) teleport the demon to a containment circle a couple miles away if you're not sure about the strength of your containment circle, b) teleport the demon to the middle of a church currently maintaining a violently strong and tempered Dominion aura (see Pax Dei, about tempering auras), c) if it's a demon that can't fly you could teleport it to the middle of the sky and watch it go splat, or d) put it in the home of one of your more detested enemies (take that, unseelie Prince of Swords!).
Those more paranoid (or knowledgeable perhaps) about what the demon could do with touch range on the spellcaster might consider inventing and casting a version of Wrench The Fallen Angel with range Arcane, adding 5 magnitudes or 25 levels to the difficulty, making it a level 65 Rego Vim spell. This is for people who have either got stacked affinities (Rego, Vim, Teleportation, & Demon Affinities, or more) and high arts, or those archmagi who don't think they have challenging enough projects in the lab and plenty of lab assistance (never try to invent a spell that would take more than 2 seasons - general guideline). To invent that version of the spell in 2 seasons would require a 98 lab total...
Moral of the story? Summoning demons Hermetically is a lot more safe than doing it the dark way, but nowhere near safe unless you are dealing with a weak demon and have overwhelming power - especially in the art of Vim (the art that gives you Twilight).
But what do you do with a demon in a circle, that can't get out but is probably screaming a large and creative list of obscenities? You could just keep casting Demon's Eternal Oblivion at leisure - if it really can't get out then I wouldn't make any of these rolls stress even if you were to do so. What would remain would be a pile of vis that would likely be non-flawed. Or, we can design another semi-permanent fate which could be worse than going back to Hell....
The most interesting idea for dealing with demons in Ars Magica is quite simple: Trap it in an object and make sure that it doesn't get out. Similar to the old genie in a bottle, this has a rich tradition in Medieval history as being something done to ghosts and other strange entities... so why not demons?
The idea of the demon trapped in the object forever, only temporarily allowed out at the whim of the possessor of the object is quite difficult to achieve. It would be quite interesting for an SG to plant a demon in a bottle at the site of an ex-temple/covenant or something, and watch the players puzzle over it. Sort of like the genie in the bottle, except even more malign. However you think about this, I suggest considering the following ritual as a starting point:
Bottling the Lowly Spawn of Satan (General, Rego Vim)
R: Near D: Instant T: Ind, Ritual
Focus: Star Ruby (+5)
This ritual traps a demon in a bottle or other such item in the possession of the ritual caster. The bottle is opened, and the demon is sucked into the bottle, which the caster then stoppers or otherwise seals. The demon is allowed out temporarily if the container is opened, to try and bargain with whomever holds the bottle to break it, granting wishes and such; if the holder of the container reseals it again, the demon is once again trapped.
A demon so contained may not use any powers unless given explicit direction or permission to by the holder of the container, after opening the container. When the container is opened, the demon cannot physically affect anything unless given explicit permission or direction to by the holder of the container.
When contained inside the closed container, the demon may do absolutely nothing - except think (usually about what it's going to try and do to whomever trapped it, when {if} it gets loose).
The stopper or other sealing method may not come open by accident unless the accident was directly caused by a person (i.e. accidentally nudging the cork so that it comes out). Often magi will use some sort of permanent enchantment to make sure nobody can open the bottle, as well as writing warnings on the outside.
Does that cover it? As for this whole business, I suspect that you could invent similar spells for ghosts, except that the difficulties might change a bit, and of course you'd be working with Mentem. Whatever other strange entities populate your game, I'm sure similar things could be done to. I suspect that faeries could have this done too, although that would start getting tricky in terms of finding arcane connections.
There has to be some interesting reason why magi would give up their souls to be taught by demons; what they have to teach and how fast they can do it is likely to be very incredible to most people. I assume that teacher demons have a disputatio score of 10 or higher, a communication score of at least 3, an intelligence of at least 3, and access to essentially limitless knowledge, skills and arts. Why would a magus even be tempted by less?
Such tutelage comes at the price of your soul, or favours, or dark worship or other such things, and it will never come cheap: Flaws like the old Tainted With Evil suddenly start appearing out of nowhere, and you find yourself unable to walk into churches, or even the Aegises of the Hearth of other covenants if you've fully given up your soul to Hell. Dark urges start appearing, and your tongue may suddenly start forking like a snake's. In short, if someone asks for your soul, it's usually a pretty good idea to say no, and follow up with everything you can muster in a fight.
So, if we all know that, why would anyone give in?
A demon would often offer a first season of tutelage for free - just to get you hooked like a drug. Maybe even more if you're skeptical. Then it would let you go about life and study without the demon, and let you go back to normal less-than-breakneck paced advancement in your magical arts, but still long enough for you to marvel at exactly how much that demon could teach you in so short a time.
Then it would start asking for small things like put this box in front of x person's door, or other little things like that, and get you back into being taught by it... then slightly bigger things, and bigger things, until you start making dark pilgrimages to places with difficult to pronounce names with no vowels, and you can't even remember what it was like to still have your soul.
Since this much is also obvious, we should never trust demons bearing gifts, right? Some poor fools would still do so, and others who have axes to grind or schemes to father would see such power as the only means towards their chosen goal, so actually seek out such demons. But of course such fools and lunatics would be few and far between, and would make poor SG villains or PCs, unless one of these lunatics were truly exceptional and capable of masquerading as normal. So, the demon would of course first start hooking you in some other guise - that of a generous elder magus, or respected scholar who feels the need to share their knowledge, and gradually suckering you into fouler deeds, and reveals their true nature only after corrupting you to or beyond the point of no return. Now, offered something free by a reputable member of the order - seems fishy? So, only do it to people who distinguish themselves, or give it as a reward to start with, and masquerade as this wise, generous person for a long time.
Suddenly, it's not a question of who wouldn't notice the trap from a mile away so much as who would the forces of Hell find worth spending that much effort on? At that level of subtlety and that long a selling time, it only becomes a matter of how long they can keep it up for. In considering exactly how long, remember that the forces of Hell have supernatural, Hermetically unstoppable powers of deceit, and centuries to practice and set up their schemes.
So who? Answer: People who could keep the secret themselves, and who would raise lineages of magi who could also do so. So the demons would likely leave the soul in the magus since taking it is obvious, despite the benefits of having security knowing that the guy couldn't take it back by finding a priest and confessing. Then, they'd sit back and marvel at how the whole thing sets a domino effect of an entire lineage of magi dedicated to the fallen ones. (The master usually looks for like-minded apprentices to mold in their own image, and 95% of the time this works perfectly.)
Now this really explains why those Quaesitores are so picky about a marched diabolist who's been dead for at least a century in your lineage, eh? Now you understand why the OoH is so paranoid about demons, and would actually consider allowing churches in their covenants.