The witness, who cannot be named for legal reasons, was at St Ninian's School in Falkland, Fife, during the 1980s.
He told the Scottish Child Abuse Inquiry on Wednesday that he had been sexually assaulted by boys at the Catholic-run institution, as well as by religious brothers John Farrell and Paul Kelly.
The two men were convicted of offences against young boys following a 2016 trial, but both deny any wrongdoing.
They were then said to have been encouraged by the men to "touch" each other, while the adults - both of the Christian Brothers religious order - would perform a sex act on themselves.
The witness, now in his 50s, told how he was sexually assaulted by a group of boys on his first day in the home.
It was heard he confided in Farrell, who began to groom him.
He said: "We were quite close, he used to give me cuddles and reassure me saying 'everything would be alright, don't worry about it'.
"I liked it because I was getting attention. It was the first time in a long time that somebody actually showed me affection - I never got any at home.
"He would give me cuddles and I would touch him and he would touch me."
He added: "In my eyes, I was having a relationship with him."
Another instance involved the witness being invited in to Kelly's room by other children, where he was then set on by them.
He said: "A group of them grabbed me, picked me up and put me on his bed.
"One of them pulled my trousers down. One of them whispered in my ear 'don't struggle' because he likes it."
It was heard the clergyman then sexually assaulted the witness.
Meanwhile, another former resident told how youngsters were "robbed" of their youth by religious brothers who molested them in care.
The man, now in his 50s, was at the Fife institution for around six months in the early 1980s.
He told how bed-wetters were targeted at night by one of the religious brothers, who pretended to check for urine.
The witness added: "We were only kids. Most of us who were turning up 12 or 13 years of age, we had our whole lives ahead of us.
"A lot of that was robbed by brothers."
The inquiry, before judge Lady Smith, continues in Edinburgh on Thursday.