The two would also confide in one another about the challenges of being a part of the royal family, Andrew Lownie said.