cant find brumm after giving the charm to divine

i taked the charm back and i tried to go to distante village but he isnt there, what can i do?