This is a difficult question to answer as farming changed a lot during the Victorian era. There was increasing mechanization and farmers specialized more and more (dairy, cereals and so on). Some farms were very small, others were very big so they would be organized in different ways. A small farm would work alongside his family - with maybe a few laborers - and they would do everything. A large farmer would need a farm manager or foreman or bailiff and, depending on his other interests or his social status, would sometimes not have anything to do with directly running the farm. A farm bailiff managed the farm and also collected rents from tenant farmers.
He was also required to be proficient in book keeping and
knowledgeable about all things farm related – right down to the
milking of a cow. Many estate owners wanted their Bailiffs to be able
to work on their own initiative as well as coming up with new money
Farming is very seasonal so they needed lots of labor for short periods of time. Wages would go up to get people to come from far away. Laborers didn't usually specialize, they did lots of different menial tasks, and children were used a lot too and would do more physically demanding tasks as they got older and stronger. One way of getting labor was with agricultural gangs. There's an article here which has quite a lot on this.
Some farmers organised their own private gangs, recruiting women and
children from the nearest village to come at busy times and work for a
few days on the farm. Others employed so-called 'public gangs'. These
were organised by 'gang-masters', who were usually unemployed farm
labourers. They recruited anything between ten and forty women and
children to work for them at a given rate of pay. They then contracted
with local farmers for their gangs to do specific jobs. The gang
master had to be able to estimate very accurately how long a job would
take. If he overestimated he would charge too much and not get the
contract. If he underestimated he would charge too little, his wage
bill would amount to more than the contract price, and he would lose
money on the deal.
The work done by the gangs varied according to the season. In winter,
when there were only a few of them at work, they were employed to
clear stones or sort potatoes. In spring their work was more varied.
Some painstakingly cleared couch grass, pulling out every root by
hand. Others spread muck, hoed, or planted potatoes. In early summer
the demand for their work reached its peak. The gangs cleared couch
from fallow fields, hand-weeded grain and root crops, and helped with
the hay harvest. Paradoxically the gangs disbanded for the main grain
harvest in August and September. Instead whole families went out to
work together. However, some gangs re-formed in October for the potato
and marigold harvest. The number of people involved in gang labour
varied according to the demand, but it was calculated in 1866 that
6,399 people had worked in public gangs in the East Midlands and East
Anglia at some time during the year.
Some farmers still had farm workers because there was always something to be done but by Victorian times there was a more obvious social gap.
in the eighteenth century farmers might well have found room in their
own homes to accommodate their labourers, but by the middle of the
nineteenth century the social gap between master and man was far too
wide for this to be an acceptable solution. Mrs Hesseltine, the wife
of a Lincolnshire farmer employing four young labourers, explained her
feelings on the subject. 'It is very objectionable having these men in
one's own house', she said in 1865, 'it is so bad for the female
servants.' Mr Hesseltine overcame the problem by making his foreman
take in the labourers
Specialist labor (e.g. carpenter, blacksmith for horseshoes and so on) could be hired in the nearest village or town - only very large farms might have their own.
Here are some other sources I used