The largest case of a nation using a proper legal system I can think of is following the Third Servile War, the famous slave revolt led by Spartacus, the Romans condemned and executed the remaining 6,000 rebel slaves that had not been slain in battle. The slaves involved were gladiators, so it was difficult to crush the revolt. Slaves are civilians, even if gladiators.
Due to the fact that slaves are not citizens and the timing of the revolt is in the late Republic, the slaves involved should be subject to the imperium. The justice system became more corrupt and the military more powerful as Rome became a dictatorship. This would mean that they would have been "tried" in a sense by a magistrate with the power to hand down military justice.
In the autumn, when the revolt was at its height and Spartacus had about 120,000 followers, the Senate voted to pass over the consuls and grant imperium to Marcus Licinius Crassus, who had been a praetor in 73 B.C. but currently held no office
There are many examples larger than this of nations using extrajudiciary justice or summary execution. The show trials during the Cultural Revolution probably would be the upper bound for largest ever death sentences. Hundreds of thousands of people were found guilty of being reactionary revolutionaries.
To place the number of 529 in a proper modern historical context, the largest simultaneous death penalty case in the Sudanese justice system during the brutal war in Darfur involved a little over 100 people found guilty of terrorist acts against the government in 2008. Following the genocide in Rwanda, the government considered executing thousands of people. Two thousand died in prison awaiting trial. The International Court of Justice stepped in and worked with Rwanda to bring to justice the worst offenders who were often hiding in other countries. The largest execution involved seven people. Considering these facts, the number seems very large.
Source: DOMAC case study 19 Sudan