Anti-tank rifles were made as a stopgap measure during and after WWI. The infantry needed something to stop a tank smaller than an artillery piece, and from further away than you could throw a grenade. There were no rockets. To give you an idea how desperate the US army was for anti-tank weapons, they tried shoving rifles and even rocks into the running gear of tanks to hilarious lack of effect (except on the rifle or rock). These tests were conducted in November 1939, two months after the invasion of Poland.
Once WWII started, plenty of these anti-tank rifles were available and tank armor was thin enough that they were still effective. Even late in the war, tanks were designed with less armor in the sides and rear to save weight and so were still vulnerable to ambush. Better than nothing!
The Germans considered anti-tank rifles to be such a threat to the sides of their tanks they added armored skirts, schürzen, to the sides of their tank hulls and turrets to protect against them. Modern "spaced armor" is used to defeat shaped charges used by many anti-tank grenades and rockets, but we know these were added to protect against anti-tank rifles. They did so because only the Soviets employed them in large numbers in WW2.
When we think of tanks in WWII we think of great big hulking Tigers and well angled T-34s. But in the early stages of WWII the situation was quite different. Tanks were thinly armored. Here's the armor thickness of some of the main armored vehicles used by the Germans during the Battle of France mid 1940.
Contemporary anti-tank weapons, too, were very small. The Panzer III's 37 mm was considered a good anti-tank gun in 1940. The Panzer II carried a 20 mm autocannon. It's only in the later war when the T-34 and KV-1 appear that guns and armor begin a rapid arms race. In this context, a 20 mm, 12.7 mm, or even 7.92 mm anti-tank rifle made sense.
Another thing to realize is the bulk of armored forces were light tanks. The Germans invaded France and the Soviet Union with mostly light Panzer I and IIs. While heavily armed and armored Allied tanks existed, particularly the French Char B1 and British Matilda II, they were very slow, and only available in small numbers.
If you imagine a tank it is a huge armored box. The chance of hitting a vital point seems incredibly small.
This is not true. With the exception of some odd experimental designs, WWII era tanks are incredibly cramped and incredibly complicated machines filled with all sorts of things that "don't react well to bullets". Every bit of available space is used to create as small a target as possible, and as light a tank as possible. Ammunition is crammed everywhere. You're very likely to hit a crew member, or a hydraulic line, or fuel line, or a fuel tank, or an oil line, or an oil tank, or a shell.
Below is a cutaway view of the US M4A4 Sherman tank. Look at all that stuff to mess up! And that's without crew.
Source: The Sherman Tank by Roger Ford pages 28-29.
For reference, here's a tour inside a T-34-85. Mind you, Soviet tanks are notorious for being built for short people, and The Chieftain is hilariously tall, but it should give you some idea.
Your round might be armor piercing-incendiary like the 14.5 x 114 mm BS Soviet anti-tank round. Now your bullet is on fire and can ignite all those flammable things inside the tank.
If you get a penetration it's not just the bullet bouncing around doing damage, but fragments of armor add a shrapnel effect. This is referred to as "spall".
Source: Think Defense: Vehicle Protection. Note: Spall liners were not used in WWII.
Because of the spall effect, you don't even need a penetration to do damage. The first field-expedient way to stop a tank with a rifle was the reversed bullet. This was a normal cartridge with more gunpowder and the bullet placed in backwards. Instead of breaking apart, the flat end would hit the armor and squash transferring all its energy into the armor. Even with no penetration, the shockwave travels through thin armor and causes spall fragments to peel off the inside armor and bounce around the tank. This was later refined to make High-Explosive Squash Head (HESH) ammunition.
Finally, for your viewing pleasure, 10 minutes of bullets impacting against armor in slow motion.
See Also