I like @Schwern's answer for its thoroughness with which the international report has been perused. Nevertheless, it seems as if some detail that is in my source isn't as clearly brought out there.
My source is Ryotaro Shiba's 'Clouds Above the Hill, Vol. 2'. I've previously commented on Shiba's accuracy here. The author does comment and generalize more than in most narratives, but these are generally based on voluminous research.
The "Japanese scare" had actually began before the Dogger Bank and the North Sea:
From the time they left Liepaja, not only Rozhestvensky but the men under him too were possessed by a delusion. The rumor was that a squadron of Japanese destroyers was lurking in the Denmark Strait to ambush them. The idea was patently ridiculous, as a cursory look at Japanese naval strength would have shown. Was it even possible to send destroyers weighing barely 200 or 300 tons all the way to the North Sea in Europe? That would have necessitated taking along an auxiliary ship for repairs, as well as a couple of protected cruisers. Japan didn’t have that kind of strength.
But even before engaging the enemy, the Baltic Fleet was dominated by fear of Japan. If anyone could pull off such a feat, they felt, it was Japan. Above all, Rozhestvensky himself was convinced of this. He and his aides had no doubt that at the very least Japan must have laid mines in the Baltic Strait along the southern tip of Sweden. ...
On the evening of their second day out, as they neared the suspected danger zone, Rozhestvensky gave orders to prepare for combat and instructed everyone to sleep with their clothes on. All the ships’ guns were readied for firing on an instant’s notice.
... Barely three days out of Liepaja, he issued orders so incomprehensible that they seem designed for the sole purpose of fomenting panic. “Everyone sleep in your clothes. All ships be ready to fire at a moment’s notice.” His actions at that point were not those of a commander but rather of someone out to create chaos.
And regarding the origin of these rumours:
Nobody knows who spread this delusion through the fleet. The probable source of the rumor was the Russian naval command in St. Petersburg. The Russian Empire’s strategic center, which should have been cool rationality, believed it possible. Rozhestvensky himself, having transferred to the fleet from a building in naval headquarters, was a believer.
After they passed into the Skagerrak, this was enhanced by further reports brought in by Russian intelligence:
This expeditionary fleet of a scale unmatched in history saw Denmark’s northern tip, Cape Skagen, on October 20, six days out from Liepaja. The entire fleet anchored off the cape, and coaling got underway. For the crews, this was backbreaking labor.
The skies were clear. At three in the afternoon, a Swedish steamboat drew near and signaled that it had an “important communication.” This was a steamboat employed by Russian intelligence, bearing important information. Rozhestvensky took the message, which turned out to be vague. “A threemasted schooner has set sail from an inlet. Highly suspicious.”
This was instinctively turned by Rozhestvensky into something far beyond what it was. It is, perhaps, noteworthy that Rozhestevensky had gone through similar fantastical leaps before:
During a court council at the start of hostilities, Rozhestvensky stressed the importance of inspecting every foreign commercial ship that entered the ports of European Russia. He refused to entertain any other opinion on the matter. His reasoning was that Japanese ships were sneaking into the ports disguised as foreign commercial ships. When Witte, who was present at the meeting, heard this, he thought, “This man is either an idiot or a coward.” In his memoirs, he wrote, “I could only marvel at him.”
“It must be a Japanese spy ship, gone out to inform a flotilla of torpedo boats lying in ambush somewhere.” Rozhestvensky’s imagination was wilder than that of any writer of fiction. The idea that Japanese torpedo boats would be lurking in the North Sea was unrealistic, but even if it were true, they would hardly use a sailing vessel to convey an urgent message. Rozhestvensky’s imagination wasn’t grounded strongly in reality. He possessed great pride, but it’s possible that excessive pride derives from morbid fear. ...
Rozhestvensky did just the opposite, putting his fear on display to the entire fleet by staging a show. He issued orders that all gun muzzles be turned on every passing ship. The fleet was not out on distant seas, but was approaching the North Sea, which thronged with traffic. ... Commercial fishing boats of each of those countries were sure to pass in the vicinity of that great fleet of imperial Russia. Every time one did, the fleet’s warships were to reposition their guns and train their sights on it. ...
... If anyone had cared to conduct an experiment in instilling an entire fighting force with mass terror, this method of Rozhestvensky’s would have been just the way to go about it.
“Everybody’s nerves were keyed up.” So wrote Engineer Politovsky. ...
But Rozhestvensky took his personal fear as the script for a show he staged using the entire fleet. They were about to enter the North Sea. The crews were sleepless, on the lookout for phantom Japanese torpedo boats. Every gun was loaded, and every gun barrel swung in unison, following each new craft that came along. “Even if we came upon a small lighter, our destroyer went off after it like blazes,” recorded one witness.
Further, the natural world heightened the sense of dread. As an idea, this is what the moon on the North Sea looks like/illuminates (taken in the Central North Sea, 2013 July):
The viewer is lucky to see the horizon; clouds show up, but the night is dark. Now, for these sailors, this wasn't the case as it was a night of fog. I've also added an image of day-time fog at sea (taken about midday, not that it matters, Central Atlantic, 2016 May):
... That night there was a dense fog, as thick as mud. As the fleet pushed on through the murky fog, each ship in turn sounded its foghorn to prevent collisions. ... The sounds heightened everyone’s fear.
The wireless communication system that enabled ships to trade information served to intensify the climate of fear all the more. Out of an abundance of caution, every ship sent out a stream of messages, reporting weird goings-on that were mere hallucinations, churning out unsubstantiated reports as if they were pelting each other with stones.
These crews were driven to the limit of human endurance by natural phenomena that their commander-in-chief allowed to heighten everyone's sensibilities. The dawning of the next day was a relief, but one all too short:
When the sun rose on the morning of October 21, the fog had lifted. At last able to verify the state of their fleet with their own eyes, the men felt relieved. Some whispered their fear of the night to come. Rumor had it that one of the captains had gone insane.
Night fell. Ever since evening, the wind and waves had been high. The fleet steamed ahead, washed broadside by waves. Before nine o’clock, a shocking radio message came in, piercing the men’s hearts. “We are being chased by Japanese torpedo boats.”
The message was from the Kamchatka, the repair ship. Named for the peninsula that the Russian Empire had annexed in 1707, the little boat was loaded with equipment, shipbuilders, and engineers to carry out repairs as needed on the rest of the fleet. The Kamchatka belonged to the First Cruiser Division in the van of the fleet but had been slowed by engine trouble and so lagged behind, bringing up the rear all alone. This must have inflamed the crew’s terror all the more.
From here on, there are relatively few additions to the @Schwern's answer which I've highlighted (italic). The rest is to add the backstory in case not all answers are read.
The Suvorov was steaming through wind and waves when the Kamchatka’s startling report came in, spurring Rozhestvensky to an immediate response. He ordered all battleships cleared for action and instructed the Kamchatka to keep him fully informed. ... the radio operator kept up a strenuous tapping and got the message through. The reply that came in was way off the mark, a fact attributable less to the state of the equipment than to the psychological stress of the Kamchatka’s captain.
“They are coming at us from all directions,” he reported. Asked how many enemy ships there were, he replied, “Eight torpedo boats.”
The reaction to such news again comes down to Rozhestvensky's personal character:
Rozhestvensky never doubted it. A moment’s calm reflection would have shown him how preposterous the claim was. Even if Japan had somehow managed to send a squadron of torpedo boats all the way to the North Sea in Europe, why deploy it against a mere repair ship? Why surround that ship with eight boats and attack it from all sides?
At the end of his exchanges with the Kamchatka, Rozhestvensky advised the repair ship to change course and get out of the path of danger. “When you have evaded the enemy, advise us of your longitude and latitude, as well as your course.”
But the Kamchatka’s response was: “Afraid to reveal.” They feared the worst if enemy torpedo boats overheard them radioing in their position. ...
At eleven o’clock at night, the Suvorov again sent out a wireless message. “What is happening? Do you still see Japanese torpedo boats?” The radio waves flew through wind and rain, but no immediate response came from the Kamchatka. ...
At length, there came a chastened reply. “No sign of them.”
Once again, the story hinges on a point where everything could have ended without trouble, but the commander-in-chief let his "Japanese dreams" take him away (much like the original quoted measure for protecting Russian harbours):
Rozhestvensky was relieved. He should have let the matter rest, but his overactive imagination lit up the dark North Sea bright as day. In the world of his imagination, eight Japanese torpedo boats were coming after the Baltic Fleet, plowing through the waves. He gave the order to prepare for action. His move was based on peculiar reasoning: “Kamchatka reports no sign of the enemy. That just goes to show the enemy has given up on the Kamchatka and is coming after the battleships.” If there is a mental disposition that sees things only through the prism of fear, then Rozhestvensky’s disposition may well belong in that category.
Time ticked away, and the great fleet steamed uneventfully on across the dark North Sea.
But the seamen in the fleet hunched sleepless at their posts. Novikov-Priboy wrote that although it was permissible for some crew members to sleep, few if any did. The captain’s psychology of fear had infected his entire fleet.
The crew were tired and on edge through the same psychology that had affected their captain. Incidentally, Novikov-Priboy wrote a fictionalized account of the battle later, 'Tsushima', having lived through the adventure on the Oryol.
In the dead of night, rain came riding on the wind. “If only there was a moon!” The men cursed the weather. They were exhausted from fighting with their own imaginations in the pitch dark. ...
Regarding the signal flare mentioned in @Schwern's reply as "green" we find the major difference in these accounts:
Shortly after one in the morning, a tricolor signal flare went up in front of the Suvorov. After the war, a survey to determine which British fishing vessel had sent up the signal flare concluded that no signal had ever been released. In any case, the fleet was in an abnormally heightened state of nerves, and so the point was never clarified. What is clear is that the Suvorov, with Rozhestvensky aboard, turned on its searchlight to sweep the darkness. Given everyone’s state of mind at the time, this was equivalent to an order to start fighting. The captain of every warship must have gasped in surprise. Aboard the Suvorov, the bugle call to prepare for action rang out. Rozhestvensky issued battle orders to the entire fleet.
Then commenced this 'battle' against the targets that came up in the searchlights:
Their opponents were trawlers. The searchlight caught a single-funnel fishing boat in a light so bright that “the red- and black-painted hull stood out clearly,” wrote the Suvorov’s ship engineer Politovsky. It was a British fishing boat. The fleet as a whole, however, mistook it for a Japanese torpedo boat and let fly a thunderous cannonade.
“At the time I was on the fore bridge,” wrote Politovsky to his wife, “but my ears were so deafened by the roar of the cannons and my eyes so blinded by the gunfire that I could not stand it and bolted below, hands over my ears, to watch the spectacle from the upper deck.” ...
The confusion was so great that the Russian naturally took there to be more than just Japanese fishing boats about:
The “battle” continued for a dozen minutes, and then one of the ships located another supposed enemy: “Japanese armored cruiser sighted.” And so the Aurora (6,731 tons), a member of the Russian First Cruiser Division, came under concentrated friendly fire. Many of the shells scored bull’s-eyes. By the time the Aurora radioed in the plaintive message “We are hit,” the damage was done. There were four perforations above the waterline, the funnel was shot off, the chaplain had lost a leg, and the chief gunner was wounded.
By then, Rozhestvensky had realized the strangeness of this naval “combat” and issued a ceasefire, but the noncommissioned officers and crew manning the guns, too excited to control themselves, kept right on shooting. It was a long time before the sea regained its serenity.
So, the real answer to the OP's question is that there was no credible intelligence, only garbled messages which let the untamed fantastical imaginations of Rozhestvensky guide him to a world of madness that he was only too suited for. It perhaps exacerbated matters that the admiral was incredibly unfit for command, with the majority of his command-time spent in coastal positions.
Shiba references Witte, the former foreign minister, using the following observation about Rozhestvensky which makes how the above events came about only clearer:
“He may be a terrible coward.”