While armed merchantmen weren't particularly effective at actually fighting U-Boats, the mere act of arming them had a tactical and strategic effect. The Battle of the Atlantic could be won by hampering U-Boat attacks as much as sinking U-Boats.
U-Boats are not really submarines, they're submersibles; they spend most of the time on the surface. Underwater they're slow and half blind with limited air and battery life. They prefer to attack on the surface, especially earlier in the war before Allied air power became overwhelming. They're faster on the surface, and it saves the battery charge. Torpedoes are very expensive, and they carry a limited supply, so the deck gun is preferred to maximize their patrol time.
On the surface they can post multiple lookouts on top of the sail giving them more eyes and the higher position means they can see further. Submerged it's just one person occasionally looking through the periscope at a very low vantage point.
U-Boats are small and fragile. Any hit might disable their ability to dive rendering them a sitting duck on the journey home. Even an old 3" gun mounted on the back of a merchantman was a threat.
If a U-Boat captain thinks the merchantman is armed they may be more cautious. They may dive and approach underwater. This drains their batteries leaving them with less time for action. They're very slow and can't see as well. A lone, and presumably fast, merchantman might dodge or outrun a submerged submarine.
They have to expend a precious torpedo, rather than cheaper and more numerous deck gun ammunition. Even if the merchantman is sunk by that torpedo, its armaments made that kill more costly to the U-Boat. It will have to sail home for resupply earlier, or risk a surface action later on for a lack of torpedoes. At the scale and desperation of the early Battle of the Atlantic, such Branninganian tactics, literally make your enemy run out of ammunition, were a win for the Allies.
Arming merchants had a severe downside for the merchantmen themselves. It made them, in the eyes of the attacking U-Boats, combatants.
In both wars, particularly WWI, the U-Boat campaign began by respecting the cruiser rules. Unarmed civilian merchants were to be stopped, searched for contraband if neutral, and the crew allowed time to take to lifeboats before sinking. This obviously put the U-Boat at risk, but it was the agreed upon rules of war. To a sailor, the sea is more the enemy than the enemy.
To not follow the internationally agreed upon prize laws was to be branded pirates. This is one of the justifications used by the US to attack U-Boats while officially neutral.
The British began arming their merchantmen with old naval guns. Rather than having untrained civilians messing about, the guns were crewed by Royal Navy and Royal Artillery gunners. The rest of the Allies soon followed.
Germany complained they were no longer armed civilian vessels subject to prize laws, they were combat vessels crewed by soldiers. This is one of the justifications for unrestricted submarine warfare: approaching a ship to stop and search it was simply too dangerous for a U-Boat.
Q-Ships, while technically not a violation of the cruiser rules, made the situation worse. These were merchant ships outfitted with hidden guns and sailing alone to act as traps. When the U-Boat approached to investigate they'd wait until it was at point blank range, unmask the guns, fly up the naval ensign, and open fire.
Such false flag ruses at sea were a common and accepted part of naval warfare; so long as the ship raised the proper flag before opening fire it was considered a clever and just tactic. However, unlike a proper warship a U-Boat is extremely vulnerable in a surface action against even an unarmored merchantman. Q-Ships exploited the cruiser rules in a way that left U-Boat captains cautious about approaching any seemingly unarmed merchant.
Q-Ships, while not particularly successful at sinking U-Boats, did lead to the unraveling of the cruiser rules, and were another justification for unrestricted submarine warfare.