This inexplicable visit is not mentioned in any English sources (had they any knowledge of it?) as the ASC was totally and strangely silent for the year entry 1064.
As the Norman sources were written after the events of 1066, their interpretation of what may have happened are therefore suspect, probably twisted out of all proportion with propaganda designed to airbrush Harold from history.
Certainly Harold seemed to be on a personal mission- Edward didn't have the power to order him to stop (though in the Bayeux Tapestry a crestfallen Harold is afterwards seen lowering his head to Edward, as if to say "I told you so"?).
Harold set out to sea from his manor at Bosham and is seen in the Bayeux Tapestry boarding his ship with hunting birds etc, but was shipwrecked on the treachorous coast at Ponthieu in a storm, despite the expertise of the English seamen.
Guy of Ponthieu (who captured Harold and his retainers on the shipwrecked coast) and Harold were previously acquainted- in 1056 on the continent in the presence of Baldwin V... and only the Normans say that Harold was in any danger!!!
Another noted Englishman- a certain Hereward (not ‘the Wake’ but exiled son of a king's thegn) similarly fell into the hands of one of Guy's neighbours (Manasses of Guines). When he established his identity, he was released and treated honourably.
Blown off-course in a storm & wrecked upon the coast notorious for this- Local Count Guy (William's former foe but now his vassal since the vicious battle of Mortemer in the 1050's) claimed all booty- as permitted from wrecks- from such "accidents" and so maybe Harold et al were victims of the false harbour lanterns that deliberately enticed ships to their doom?
Harold & his advisor/huscarl/servant retainers were fettered by Guy's men (the brutal types who would not normally even blink at torturing a hostage to exact higher ransoms) & led off to a gaol at Guy's castle at Beaurain.
But one escaped to find the ambitious & greedy William who, seeing a golden chance for personal gain(Ransom? Succession issue? Sizing up his rival?) he immediately raced to Guy and ordered his vassal to hand them all over- or maybe 'dire events' would ensue.
William had them taken to his Norman capital Rouen as his honoured "guests", where they met William's own family, watched tournaments and ate lavishly etc. It suited William to be shown as Harold's rescuer (obligation to him) and tie the powerful earl to him by even knighting him!
Harold was possibly even flattered by William's praise of his recent Welsh 'conquests' in order to win him over?
Maybe William browbeat him into more than one Brittany campaign with the Duke (wanting to witness his man's battle strengths & weaknesses for a later date?).
William used a rebellion in Brittany's east to advance to Dol, Rennes and to Dinan (to punish Count Conan II for recent raids into Normandy's Western frontier, where Conan submitted). Harold distinguished himself valiantly- once pulling two drowning Norman troops out of marshes.
William and Harold had both got the measure of their 'opposite' numbers (strengths, weaknesses, battle-prowess and character) and maybe only superficially exchanged pleasantries for a time they both by now must have inwardly known would soon drastically change?
Back at the scheming duke's main hall in Rouen(?), William then secretly had some holy relics hidden in a chest beneath a sheet...with an "oath" from Harold in mind...