The spy and the sentry

A spy wanted to enter a castle, but this castle was guarded by a sentry. Only those who knew the password, were allowed to enter. The spy hid himself in the bushes near the guardhouse of the sentry, so that he could overhear the password.
The baker approached, and the sentry called:
‘If I say 12, what do you reply?’
‘6’
‘You may pass.’
The smith approached, and the sentry called:
‘If I say 6, what do you reply?’
‘3’
‘You may pass.’
The spy concluded: ‘I know enough’
With a long detour he went back, disguised himself as a grocer and approached the sentry. The sentry called:
‘If I say 4, what do you reply?’
‘2’
The spy was taken prisoner.
What should he have replied?

I would like to thank our daughter Margreet for passing on this nice problem, which she heard from Professor Jochem Thijs. Alas he did not reply to my question if he invented this puzzle or not. If he is not the inventor, and someone knows the original source, I would be grateful.

You can check your solution here

You are welcome to remark on the puzzle: its wording, style, level of difficulty. I love to read your solution times. Please do not spoil the fun for others by listing the solution.