Brown, who had been in charge at the club for more than five years, gave his notice after what he described as ‘unfair’ treatment following Rovers’ match at FC Clacton.
Suffolk FA upheld an allegation in the referee’s report that Brown had verbally abused the official, and issued a £100 fine.
Brown appealed to the Suffolk FA’s disciplinary panel but, much to his fury, the original decision was upheld.
"I can’t work with these people anymore," he said. "I don’t expect to be treated like that. The way the Suffolk FA have handled this is not fair."
Brown said he had been applauding the players off the pitch after the match at Clacton when the referee told him he would be reporting him for misconduct.
"I asked him why he’d be reporting me and he didn’t answer," said Brown. "He also said he’d seen me in hospitality before the match, but he couldn’t have done because I didn’t attend the hospitality there. I only arrived at the ground at 2.40pm.
"I had witnesses from Clacton and the guy I was with who said I hadn’t been in hospitality."
Brown could have appealed to the main FA based at Wembley, but he said the costs involved meant it was not worthwhile.
"Even if you win you still have to pay the costs, so it’s not really an appeal," he said.
Jairo Marin, football services manager at the Suffolk FA, said: "The Suffolk Football Association charged Mr Brown with misconduct in accordance with FA regulations upon receipt of the referee’s misconduct report in relation to the match against FC Clacton.
"In line with standard procedure Mr Brown had the option to appeal this decision.
"Suffolk FA’s disciplinary commission heard the case and finding the evidence against Mr Brown proven, fined him £100. Mr Brown was advised at the time he had the right to appeal the decision to the Football Association."
Brown added: "I’m sorry for Haverhill. I’m proud of what we’ve achieved at the club and I’ve thoroughly enjoyed it.
"When I came the club was in debt, but it is now financially sound and I’ve overseen the move to the new ground."
Haverhill are due to hold an extraordinary general meeting to elect a new in due course.
First-team manager Peter Betts said: "The club is extremely disappointed to lose someone of Steve’s calibre. People don’t always see the hard work that goes into running the club, but what he has done for the club is phenomenal."
Credit: Cambridge News