First, I have a few questions, Is this your first horse or are you an experienced horseman? Secondly, can you touch the horse all over, petting him and rubbing him, or is to scared for that also?
The two or three people holding your horse while you try and get a halter on him, is probably freaking him out to begin with. He has already had bad experiences with people, and two or three people surrounding him, trying to hold him are really making a bad situation worse.
Abused horses take a lot more time and patience to train than one that has not been abused. It will take him a while to let his guard down. You will have to earn his trust. When you do get a halter on him, spend a lot of time, just petting and brushing him, letting him know that you're not trying to kill him.
Also, I hate to tell you this, but every time you give up while he is still acting out, he will be even worse next time. It could even be dangerous, if you don't know what to do, in any given situation.
I want to recommend a video for you to check out. It is by Ray Ainsworth. He is a very good horse trainer. As a matter of fact, he is a good friend of mine. You can get the video from his website. rayainsworth.com
He teaches you how to cure most problems, using groundwork, the correct way.
After watching the video, if you don't think you can handle this problem yourself, then you should seek out a professional trainer. But please, make sure that it is a good trainer. Some, so called trainers, can make the problem worse.
How Do I Get My Horse Not To Be Afraid Of The Halter, Etc? by: Anonymous
Make it a good experience for your horse, by offering treats (such as small carrots or cut up apples). If she runs away, just crouch down; this is supposed to stimulate your horses curiousness. If she doesnt come, just leave (if you have other horses, put the halter on them and then give them the treat). That might help. If you ever get her close enough to the halter, don't put it on. Just give her a treat and a nice rub and you should also be holding the halter (or put it on your shoulder)while you are doing this. Since she pays so much attention to the halter pretty much all the time like you said, you wouldn't have to do much to show her the halter. And also she does probably get freaked out when 2 or 3 people have to hold her down just to get a halter on. That would not be a very pleasant experience. You could also do the tellington touch (if you don't know, you should probably look it up, but I don't know if you would probably find it that easily). Do that a couple of times and then start to rub the halter on her neck and around her (don't swing it) so she gets more used to it; remeber to rub the halter on her while you are giving her a treat (or while she is chewing it) and/or while you are rubbing her. Good luck with your horse!
scared of halter by: JIm B
All horses are scared of the halter at first, they need to learn that it is not going to hurt them, some are more fearful than others that's why I start training them as soon as possible, and I don't give up. I don't recommend treats but whatever works for you, I put the halter on, take it off, put it on, and so on for days, weeks if need be until he expects it, usually before feeding him, until he is comfortable with it and even expects it. the same with saddle and so on. I have trained horses for a long time and love to work with them, and even before trying to work with halters, saddles, etc. I work on their trust of me. so when I do try to put these strange objects on them I am not really scaring them too bad. Every time i feed i start by picking up feet, rubbing them down, combing their tail, while they are feeding, at first they don't like it but they want the feed and eventually put up with this bothersome person. I tell people that if you work with a horse enough, he will never buck, and have proved it time and again.