It's not just an actor-stature thing though. Scaramanga's eponymous firearm is a modest pistol that collapses into a collection of small discreet components (similarly, his plane is constructed from his car... and then suddenly shrinks to a model in a cunning cut that allows it to actually take off). To make up for having such a poky signature weapon, Scaramanga has taken delivery of a huge great laser cannon. No wonder he's so glad to see Bond in the third act of the film as he gets his one and only chance to actually use it. Indeed the whole megalomania sub-plot involves a power station built into a pair of islands - but, perversely, only run by a single man (the virtually mute Sonny Caldinez) and functionally dependent on a cassette-tape-sized component, the 'Solex agitator'.
Bond overcomes Scaramanga by looking past the optical illusions and staging - literally, climbing back through the stage scaffold to profitably replace his own waxwork mannequin. The resolution of the thriller is in his assimilation of the dizzying disproportions throughout the film. This is, in fact, the essence of Bond - keeping a grip on reality in the midst of the megalomaniac's altered paradigms as the route to triumph.
*I have no evidence to assert this whatsoever!