This is probably a very debatable question, but I think I can make the argument, with good historical backing, that it was the non-violent protests that were most effective in what progress was made in the Civil Rights movement.
Firstly I make this argument in deference to the leaders on the ground. A reading of Freedom Summer, by Bruce Watson* shows that the leadership of SNCC had extended, passionate discussions about whether to use "self-defence" or entirely passive resistance tactics. Stokely Carmichael, future leader of the Black Panthers, unsurprisingly was the lead voice for self-defense. This leadership was strongly southern blacks (unlike Carmichael), who knew exactly what kind of a buzzsaw they were walking into. That they settled on the tactic they did has to at least show that this was what they saw as the most effective tactic at that time. All of these people were in a far better position to judge that than any modern person who has not lived through what they did.
Secondly, lets look what happened with a pure self-defense approach. We can't look at that in isolation in 1964, but we can in 1921. On May 31st of that year, in response to an imminent lynching action, a similar discussion went on the black community of (north) Tulsa. In this case, the case of those arguing for active resistance and self-defense won the day. Guns were gathered, and volunteers set off for the jailhouse to offer their help to the Sheriff. There they dispersed a mob of 1,000 whites besieging the sheriff. So far so good.
The angry mob of whites put out the word that armed black folk were taking over town, the armory was raided, and soon the black community found itself in a pitched battle over the railroad tracks separating the two communities. The north Tulsa community fought bravely, but they just didn't have the numbers. At about 5 AM the tracks were overrun, and the white mob stormed into North Tulsa, and burned it to the ground. By the time the National Guard restored order the next day, hundreds were dead, over a thousand injured, six thousand "interned", and here's what the richest African-American district in the nation looked like the next day:

This wasn't the only such incident of African-American communities attempting self-defense, but it is probably the largest scale one. The people making decisions in the Civil Rights movement knew about these incidents, and others like them. They knew that if you don't have larger numbers, and/or political control, this is what the result of non-passive resistance looks like for you.
* - It is simply not possible for me to recommend this book highly enough.