What intelligence did the Russians have that the Japanese had either torpedo boats or mines in the North Sea, and what was the source of that intelligence?
I don't have the book referenced in the article, but the report from the International Commissions Of Inquiry is online. It mentions several "reports" (really rumors) in more detail.
It appears, from the depositions made, that, from the time of the departure of the squadron from the roads of Réval, Admiral Rojdestvensky had had extreme precautions taken by the vessels placed under his orders, in order that they might be fully prepared to meet a night attack by torpedo boats, either at sea or at anchor.
These precautions seemed to be justified by the numerous reports received from the agents of the Imperial Government on the subject of hostile attempts to be feared, which in all likelihood would take the form of attacks by torpedo boats.
Moreover, during his stay at Skagen, Admiral Rojdestvensky had been warned of the presence of suspect vessels on the coast of Norway. He had learned, also, from the commander of the transport Bacan coming from the north, that he had seen on the previous night four torpedo boats, carrying a single light only, and that at the masthead.
Then the transport Kamchatka falls behind at night with engine trouble and imagines a passing ship is a torpedo boat. She opens fire and radios that she's under attack.
The last vessel which passed the Zero was, from his description of her, the Kamchatka.
This transport, which originally was in a division with the Dmitri Donskoi and the Aurora, was, therefore, left behind and isolated about 10 miles to the rear of the squadron. She had been obliged to slacken speed in consequence of damage to her engines.
This accidental delay was, perhaps, incidentally the cause of the events which followed.
Toward 8 o'clock in the evening this transport did, in fact, meet the Swedish vessel Aldebaran and other unknown vessels and opened fire on them, doubtless in consequence of the anxiety inspired in the circumstances of the moment by her isolation, the damage to her engines, and her small fighting value.
However this may be, the commander of the Kamchatka, at 8. 45 o'clock, sent a message by wireless telegraphy to his commander-in-chief regarding this encounter, stating that he was "attacked on all sides by torpedo boats."
This just fuels Rojdestvensky's and the fleet's paranoia as they sail on into the night thinking they're being trailed by torpedo boats. He orders weapons free.
In order to understand the effect which this news had on Admiral Rojdestvensky's subsequent decisions, it must be remembered that, in his estimate, the attacking torpedo boats, of whose presence, 50 miles to the rear of the division which he commanded, he was thus, rightly or wrongly, informed, might overtake and attack him about 1 o'clock in the morning.
This information led Admiral Rojdestvensky to signal to his ships about 10 o'clock in the evening to redouble their vigilance and look out for an attack by torpedo boats.
On board all the ships, moreover, the standing orders of the Admiral laid down that the officer of the watch was authorized to open fire in case of an evident and imminent attack by torpedo boats.
If the attack was from the front he was to open fire on his own initiative, and, in the contrary case, which would be much less pressing, he was to refer to his commanding officer.
As they're passing the fishing fleet, a green flare appears.
The direction in which this division was sailing led it nearly toward the main body of the fleet of trawlers, round which and to the south of which it would therefore be obliged to sail, when the attention of the [p934] officers of the watch on the bridges of the Souvoroff was attracted by a green rocket, which put them on their guard. This rocket, sent up by the “admiral" of the fishing fleet, indicated in reality, according to regulation, that the trawlers were to trawl on the starboard tack.
When they spot an unknown vessel apparently heading towards them with no lights, they opened fire.
Almost immediately after this first alarm, and as shown by the evidence, the lookout men, who, from the bridges of the Souvoroff, were scanning the horizon with their night glasses, discovered " on the crest of the waves on the starboard bow, at an approximate distance of 18 to 20 cables, "a vessel which aroused their suspicions because they saw no light, and because she appeared to be bearing down upon them.
When the suspicious-looking vessel was shown up by the searchlight, the lookout men thought they recognized a torpedo boat proceeding at great speed.
It was on account of these appearances that Admiral Rojdestvensky ordered fire to be opened on this unknown vessel.
With the fleet in such a state of panic, everyone started firing and there was no stopping it.
The Admiral then made the signal to the squadron "not to fire on the trawlers."
But at the same time that the searchlight had lit up this fishing vessel, according to the evidence of witnesses, the lookout men on board the Souvoroff perceived to port another vessel, which appeared suspicious from the fact of its presenting the same features as were presented by the object of their fire to starboard.
Fire was immediately opened on this second object, and was, therefore, being kept up on both sides of the ship, the line of ships having resumed their original course by a correcting movement without changing speed.
And why would it be considered so reliable that the Russian navy would open up on a variety of ships in the North Sea thinking they were Japanese?
The Wikipedia article seems to cover it. But I think the psychological threat of torpedoes and torpedo boats to early 20th century navies cannot be understated.
In the 19th century "torpedoes" as in "damn the torpedoes" meant mines. Mines were terrifying enough to navies, a hidden threat that could lurk almost anywhere and blow up a capital ship without warning. The mere possibility of mines has made any number of fleets shy away from an attack or pursuit.
20th century torpedoes are "motor torpedoes", basically self-propelled mines, starting with the Whitehead Torpedo in 1866. Quickly naval planners realized that a flotilla of small, cheap torpedo boats could counter expensive, heavily armored, and otherwise impervious capital ships. Even if the whole flotilla was lost, it was worth it if they sunk one battleship.
This was so feared they created an entirely new class of ship, the Torpedo Boat Destroyer, just to counter it. Larger, faster, more heavily gunned than a torpedo boat, these grew and eventually replaced the "cruiser" as the workhorse of the fleet becoming the modern Destroyer.
Now any nation with a large capital fleet has to contend with not only losing a ship by running over a single hidden mine, but now losing the whole fleet to swarms of small, fast boats firing torpedoes. This upended the slow, stately pace of capital ship warfare they'd trained for, with careful formations, strategic maneuvers, and chewing away at each other in gunnery slugfests.
Even if they missed, dodging torpedoes would break up their careful formations and cause the slow, lumbering capital ships to turn away from the fight. This happened famously to the Yamato during the Battle off Samar as she had to run away from the battle to avoid torpedoes from a US destroyer taking her out of the fight.
Add to this that the Japanese already had a reputation for surprise attacks, having opened the war with a surprise night torpedo attack on Port Arthur taking two battleships out of action.
The idea that the Japanese could send a handful of these cheap, small hard to detect torpedo boats which could suddenly sink a battleship without warning was terrifying (and very, very overblown).
Now I'm going to get into a bit of speculation.
The Russian navy did not have the globetrotting tradition like the US or Royal Navy. Russian Baltic fleet was a coastal fleet used to being in safe, known waters. Now it's being asked to travel halfway around the world, exposed to who knows what, for months! Spotter aircraft were not available, and radios were new and poorly understood. They had to rely on visually spotting threats, their world of information was restricted to just a few dozen miles and a hazy horizon.
There's a lot of ocean. One can imagine a lot of threats locked up in a steel hull for weeks on end.
While traversing the narrow waters between Denmark and Sweden, the Russians imagined this to be a perfect place for a surprise attack. They had numerous false alarms about mines and torpedo boats, and one incident of firing on fishermen who were just relaying messages from Russia. Rather than settle his men, Rozhestvensky stoked their fears with orders saying no vessel of any sort should get among the fleet.
They were already on edge.
That's why the hysteria. Why did they take this seriously? Here's my best guess.
Navies are very conservative, necessary when you're dealing with expensive assets that take years to build and can be destroyed in an instant. Navies plan based on capabilities, not probabilities. If the enemy is capable of an attack, you must plan for it. One of the greatest failures of naval planning was ignoring that the Japanese were capable of striking Pearl Harbor, even though most thought it very unlikely.
The Russians felt that if they could send a capital fleet halfway around the world, the Japanese could do it with some torpedo boats, too.
As for the mine field, mine fields are cheap and easy to lay in secret under the guise of, yes, a fishing boat. The idea that the Japanese might have mined the Russian Baltic fleet's route was not so far fetched. The idea that they'd recklessly endanger international shipping 30,000 km from the war zone... that was a bit far fetched.
Mix in that the British had a defense treaty with Japan, and you could imagine Japanese agents and boats operating out of secret bases in Britain.
Add in healthy doses of paranoia, isolation, inexperience, and a new and poorly understood threat, and suddenly you're shooting for 20 minutes at a fishing fleet.