Whitney Houston was a tabloid obsession for most of her three decade career. But six years after she died in her bath aged 48 from a combination of heart disease and cocaine use, she’s proving equally irresistible to film-makers seeking to make sense out of the drama of life – and to use it as a prism through which to view the 1980s and 1990s, a period when she was touring the globe, shifting records by the million and paving the way for a generation of black female singers.

Actress Angela Bassett, who co-starred with Houston in 1995 film Waiting To Exhale, turned director in 2015 to make Whitney, a biopic in which Yaya DaCosta played the singer and Arlen Escapeta her much-maligned husband, rapper Bobby Brown. In 2017 British documentary-maker Nick Broomfield pitched in too. Having already made films about Kurt Cobain and murdered rappers Christopher "Biggie" Wallace and Tupak Shakur, he added Houston to his list of tragic pop culture subjects with Whitney: Can I Be Me. It delved into Houston’s prolonged and sustained drug use, and the fallout from her lesbian relationship with long-time friend and confidante Robyn Crawford.

Now, fresh from its UK premiere at the Edinburgh International Film Festival, comes film number three: another documentary, this one from Oscar-winning Scot Kevin Macdonald, but this time made with the full co-operation of Houston’s surviving family members – her brother and half-brother, Michael Houston and Gary Garland; her mother, former 1960s soul singer Cissy Houston; and Bobby Brown himself. Packed with interviews and archive footage, much of it never seen before, it doesn’t shirk from examining Houston’s drug use but it asks even the most sceptical of viewers to re-assess the singer and her legacy. And, sensationally, it shines new light on her troubled life by unearthing a dark family secret never previously discussed.

I meet Macdonald in an Edinburgh hotel on the day of the premiere. The Glasgow-born 50-year-old is best known as a director of dramas such as The Last King Of Scotland, State Of Play and How I Live Now, but much of his early work was documentary-based and his film about Houston follows a similar 2012 film about Bob Marley. Like Whitney, it was made with the full co-operation of the reggae legend’s estate though Macdonald insisted on full editorial control, as he did here.

“The arrangement was that I could do whatever I wanted and I’d have to show them some cuts and they could comment, but I wouldn’t have to pay attention to what they said,” he tells me. “And they had a healthy attitude which was that there was no point them producing another puff piece. Whitney’s reputation was so in shreds and tatters anyway, so what are you protecting?”.

Of course Bob Marley is one thing, Whitney Houston quite another, especially for someone whose music tastes in the 1980s and 1990s probably didn’t stretch to I Wanna Dance With Somebody or Saving All My Love For You. So how did Macdonald view Houston when she was in her globe-straddling prime?

“My attitude was probably pretty dismissive,” he admits. “I had the attitude probably a lot of people of my generation had who had lived through seeing her, even if very tangentially out of the corner of their eye, populating the front pages of the tabloids. Her foul mouth and crude and crass utterances here and there, her endless fights with Bobby Brown. I just thought she was a sort of tabloid loser and someone who it was very hard to have any sympathy with.”

So he was initially sceptical when first approached by the Houston estate to make the documentary. But three things helped change his mind. First, he met with Houston’s long-time agent at America’s Sundance Film Festival. “She said ‘I really want to make this film and I’ll help you in any way and the reason I want to make it is because I loved Whitney but I never understood her and I really want you to try to understand her for me’. That kind of personal appeal and also the mystery of it – how can somebody know someone personally and professionally for 25 years and not feel like they understood them? – was intriguing”.

Then he read a long article in The New Yorker magazine about Houston’s now legendary rendition of The Star-Spangled Banner at the 1991 Super Bowl. The game was played 10 days into the first Gulf War and national feelings were high. But instead of belting out the anthem Houston had it slowed and radically re-arranged to the point where the orchestra was said to have hated it. The effect was to turn it into a gospel-flavoured and emotionally-wrought hymn which struck a chord with the both Super Bowl crowd and, when it was released as a single, the record-buying public.

Finally, and most simply, Macdonald sat down and listened to Whitney Houston’s music and found himself moved by it. “That’s the key to her brilliance and the thing that maybe wasn’t recognised by me when I was of the age to listen to it,” he says. “It just seemed like bubble gum pop but actually it was amazingly soulful … There definitely is a communicative, emotional, interpretive genius there. So those three things made me think ‘OK, there’s an interesting film to be made here’.”

Macdonald’s initial misgivings aside, it’s not hard to see why Houston’s life and death exert such a pull. Born in New Jersey in 1963, her godmother was the singer and actress Darlene Love and her cousins were soul superstar Dionne Warwick and her sister Dee Dee, also a singer. Aged eight Houston was hanging out with Aretha Franklin and by her mid-teens she was singing backing vocals and touring nightclubs with her mother. She later worked as a model before signing to Arista in 1983.

She released her self-titled debut album, Whitney Houston, in 1985, followed it with Whitney two years later and by the time of 1990’s I’m Your Baby Tonight had won a clutch of Grammys and Brit Awards and become the first (and still the only) recording artist to have scored seven consecutive number ones. Like Madonna, it only took one name to identify her. Like Madonna, she was soon also making films, starring alongside Kevin Costner in 1992 smash The Bodyguard.

But it was off-stage that the drama lay. Throughout the 1980s Houston dated a string of high-profile men such as Eddie Murphy and Philadelphia Eagles quarterback Randall Cunningham, but at the 1989 Soul Train awards – a year after she had been booed at the same event, supposedly for sounding too white – she met rapper Bobby Brown. In 1992, they were married and a daughter, Bobbi Kristina Brown, was born a year later.

The couple’s life together became the stuff of lurid tabloid headlines, as rumours about drug and alcohol abuse swirled around them. There were allegations of domestic violence and run-ins with the police – Brown was jailed briefly at one point – and Houston’s physical appearance and often erratic behaviour was picked over and scrutinised at every turn. Not without reason, either: at a 2001 concert in Madison Square Garden in honour of Michael Jackson she appeared worryingly thin and the following year, in a primetime interview with ABC’s Diane Sawyer, she faced down the rumours that she was dying but did admit to having used marijuana, cocaine and prescription drugs in the past. Not crack, though. “Crack is cheap,” she told Sawyer. “I make too much for me to ever smoke crack.”

“I think part of what’s interesting about Whitney is that she’s so analysed and talked about,” says Macdonald, “even by people you wouldn’t think would be interested in discussing the ins and outs of her relationship with Bobby Brown. I’m always amazed about how people can talk on and on about this. They find a certain fascination in her.”

Another rumour that followed Houston was that she was gay, and had been in a relationship with Robyn Crawford. As with the gossip about her drug use, this rumour was true: Houston and Crawford had met in the late 1970s and almost certainly had a relationship. Crawford then became a confidante and companion to Houston though after years of conflict with the singer’s family she finally cut all ties in the late 1990s. In Macdonald’s film, Gary Garland doesn’t mince his words about Crawford.

“The audience can make their own mind up about Robyn,” says Macdonald. “I think most are going to feel tremendous sympathy for her when that’s said about her and they’re going to understand why her being around was so difficult, because there was such disapproval in the family.”

So how serious and long-term was the relationship? “I don’t think it went on,” he says. “I’ve seen various documents and letters which I couldn’t show in the film which show they were an item – romantically, sexually – for a few years before Whitney was famous. After that it really was this very, very close friendship and they made this conscious decision that they should break it off. Whitney liked men but she happened to love Robyn as well.”

Today, of course, such sexual fluidity is unremarkable. Where pop stars are concerned, it can even be brand-enhancing, a powerful form of cultural capital. That wasn’t true, however, in the 1980s and 1990s.

“People keep asking me ‘Was Whitney gay?’,” Macdonald adds. “Ten or twenty years ago you’d have said she was bisexual. But now I would say ‘Why put a label on it?’. As far as I’m aware, the only gay sexual experience she had was with Robyn. There may have been others, but I don’t think so. Generally she chose men.”

Macdonald tried to get Crawford to talk on camera but, although she was initially positive and in communication with him, she eventually said no. “That was a big blow. But I suppose as a consequence of that the film became more about a family story, the world of the family around Whitney and the world that she came from.”

Bobby Brown did speak to Macdonald, though. Defying his bad boy reputation, he comes across as thoughtful and likeable. Macdonald laughs when I mention it. “Whitney’s old agent, who helped me put together the interviews with the tricky people like Bobby, said to me: ‘You’re going to really like him. Bobby’s really likeable’. And he is.”

But Macdonald also found a man still deeply troubled by the deaths of Houston and of Bobbi Kristina, whose death came in circumstances hauntingly similar to her mother’s: she was found face down in her bath in January 2015 and died in hospital six months later having never regained consciousness. Again, drugs were a major contributing factor.

“He [Brown] was nervous and suspicious when he was on camera, and obviously just isn’t ready to talk openly,” says Macdonald. “He’s in denial. It’s all too painful for him, or whatever, but he doesn’t want to go there. But Bobby has been placed in this public soap opera that is their lives. He’s Bad Boy Bobby, the villain who destroyed the life and career of this beautiful princess. As the film makes clear, I hope, that’s miles from the truth. The truth is that they loved each other and she chose to stay with him. She was already way more into drugs than he was and I think it may even be arguable that he only had drugs a few times when he was hanging out with her.”

But it was towards the very end of the filming process that Macdonald really struck gold, though gold of a tawdry and distressing sort. Listening to audio of a 1989 interview Houston had done with Radio 1 DJ Simon Bates, the director was struck by her answer to the question “What annoys you most”. She said: “Child abuse”.

“That bit of interview combined with another interview where she says something similar made me think: ‘Has she suffered some trauma in childhood? Is that a way of understanding her character?’”.

Macdonald broached the subject in the fourth and final interview he did with Gary Garland. In it he tackled Garland’s drug problems – he still struggles with addiction – and in particular what it was that made him keep returning to drugs. Garland told him it all went back to when he was abused as a child.

“From that came the fact that Whitney was abused as well,” says Macdonald. “That was very, very late in the day and in fact the interview which brought all that together was with Mary [Jones], who was Whitney’s assistant, who was the only person who ever talked to her about it in any detail.”

Jones hadn’t mentioned it in Macdonald’s first interview with her – Garland’s comment that “This is a family with a lot of secrets” couldn’t have been more true – but when she knew that Garland had spoken about it, she flew to London from America for a final filming session. “This is the elephant in the room,” she told Macdonald. “This is the thing that nobody will talk about and I think they should.”

The alleged abuser was Dee Dee Warwick, younger sister of 1960s soul star Dionne Warwick and also a singer. She died in 2008, aged 66, having struggled for many years with her own drug problems. “Even Whitney’s own mother didn’t know,” says Macdonald. “Or says she didn’t know.” Perhaps predictably, nobody in the family would talk about Dee Dee Warwick, on camera or off.

Today, Whitney Houston’s artistic legacy has been reclaimed by a younger generation who come to her with no baggage and who hear only a collection of peerless pop songs performed by a woman with an extraordinary soulful voice. Meanwhile the many ups and downs of her life, her struggles with drugs, her relationships and her sad end all put Houston into the same iconic category as Amy Winehouse, like her the subject of a recent high-profile documentary.

But for Macdonald there’s a third layer to her appeal.

“From my point of view it’s because she’s a real, true tragedy in the Aristotelean sense,” he says. “She’s someone who is more beautiful, more talented, richer than almost anybody else and yet by the end of the story she’s brought so low by character flaws. I don’t think in the end you can blame anybody else but her and I think there’s a ghoulish fascination in seeing somebody brought low, and recognising our own mortality in that.”

Whitney is released in cinemas on Friday