"A big reason why we stopped it now was based on the time of the rally on Monday and the intention of some people to potentially carry out violent acts in Richmond," Jay Tabb, executive assistant director of the FBI's Department of Homeland Security, said in an event promoted by the Homeland Security Experts Group in Washington.
Seven men accused of belonging to a white supremacist group called The Base were arrested this week in separate attacks in Delaware, Georgia, Maryland and Wisconsin, officials said.
Tabb praised police work in Virginia to prepare for the demonstration, saying "they are doing everything they can to make this event and all of Monday's events as safe as possible".
Still, he said, the FBI is concerned.
"I think the FBI has a good sense of concern for Monday because of all the factors that you just talked about and because we can't explain everything and everyone. We have a certain interest in some people that we know they are for less saying that they will be there and that we have no way of predicting where the rhetoric turns to violence, "said Tabb, referring to the context of the rally: inflamed tension around the gun laws, Martin Luther King Jr. day and a historic day of celebration for the Confederates in the area.
"There is a high degree of interest in some individuals from both spectra in the ongoing debate over gun rights, ethnic and racial issues, which potentially leads to this event on Monday and could end some type of criminal activity for which they can be arrested, "said Tabb.
CNN reported on Thursday that investigators believed the Delaware and Maryland men were planning to attend the rally, according to a law enforcement official.
Tabb described the Base as a "neo-Nazi group that believes in white power and called for a white ethno-state in the United States".
He said the group recruits widely online and its central presence is in cyberspace.
"I think what we find with individuals who claim to be part of that group is that they also claim membership or have affiliations with other like-minded individuals who subscribe to a similar ideology," he said. "I have to say this: there are not many physical members in these groups, there are many virtual members".
CNN's Rebekah Riess contributed to this report.