The Justice Department has fired the No. 2 official in the U.S. Attorney’s office for the Eastern District of Virginia after he declined to lead the controversial prosecution of former FBI Director James Comey, according to multiple people briefed on the matter who spoke on condition of anonymity to speak freely.

Robert McBride, 64, a Justice prosecutor and former Navy lawyer, was brought into the prominent satellite office of the Justice Department to serve as first assistant to U.S. Attorney Lindsey Halligan and took a more prominent role as her status was in question and after a judge ruled in late November that she was not legally appointed to run the office.

McBride, a prosecutor and former supervisor in a U.S. Attorney’s office in Kentucky, had been asked in recent days to run the Comey case, and told top Justice officials he felt it would difficult to do that and also run the office, according to the people.

Halligan had also recently learned that McBride held private meetings with federal judges in the Eastern District of Virginia, according to a source familiar with McBride’s removal.