In physics, work is said to be done, if a force acting on a body is able to move the body through some distance in the direction of the force.
Here, the boat is not moving with respect to the bank and hence the work done by the man is zero.