Robertson was born Jaime Royal Robertson on July 5, 1943. He was an only child. His mother was Rosemarie Dolly Chrysler, born February 6, 1922. She was Cayuga and Mohawk raised on the Six Nations Reserve southwest of Toronto, Ontario. Chrysler lived with an aunt in the Cabbagetown neighbourhood of Toronto and worked at the Coro jewellery plating factory. She met James Patrick Robertson at the factory and they married in 1942.
Rosemarie and James Robertson continued to work at the factory where they met. The family lived in several homes in different Toronto neighbourhoods when Robbie was a child. He often travelled with his mother to the Six Nations Reserve to visit her family. It was here that Robertson was mentored in playing guitar by family members, in particular his older cousin Herb Myke. He became a fan of rock 'n' roll and R&B through the radio, listening to disc jockey George "Hound Dog" Lorenz play rock 'n' roll on WKBW in Buffalo, New York, and staying up at night to listen to disc jockey John R.'s all-night blues show on WLAC, a clear-channel station in Nashville, Tennessee.
When Robertson was in his early teens, his parents separated. His mother revealed to Robertson that his biological father was not James, but Alexander David Klegerman, a man she met working at the Coro factory. Alexander was Jewish. He became a professional gambler and was killed in a hit-and-run accident on the Queen Elizabeth Way. She had been with him while James Robertson was stationed in Newfoundland with the Canadian Army, before she married James. After telling Robertson, his mother arranged for the youth to meet his paternal uncles Morris (Morrie) and Nathan (Natie) Klegerman.