The report by the House of Bishops called for a “culture of welcome and support” for gay Christians.
But it maintained that marriage in church should only be between a man and a woman, and services should not be held to bless same-sex relationships.
The motion was defeated by the House of Clergy, following hours of debate.
The House of Bishops voted overwhelmingly (43-1) in favour of the report, and the House of Laity backed it by 106 votes to 83.
But to win approval the report had to win backing in all three houses and the Clergy rejected it by 100 votes to 93, with two abstentions.
Bishops will now have to produce a new report on the issue.
The decision was welcomed by LGBT rights campaigners, some of whom had staged a protest ahead of the debate.
Lucy Gorman, an activist and Synod member from York said: “Thank you Synod. With that vote we’ve sent a message to the outside world.”
Concluding the debate, the Archbishop of Canterbury Justin Welby, who supported the report, called for a “radical new Christian inclusion”.
He said: “There are no problems here, there are simply people.”
He added: “The current report is not the end of the story. We will – as the bishops – think again and go on thinking. We will seek to do better.
“It was right that this needs to be about love joy and celebration of our humanity, of our belonging to Christ, all of us without exception, without exclusion.”
There is no easy way to dress up what has been an embarrassing night for the senior leadership of the Church of England.
After three years of so-called shared conversations costing the church more than £300,000, General Synod has chosen not to take note of the Bishops report.
It was neither the Bishops nor ordinary members of the church (the laity) who chose to reject the report. It was the vicars, rectors and priests that decided they could not continue with the current prohibition on blessing or marrying same sex couples in church.
For lesbian and gay Christians, there is widespread rejoicing. But conservative evangelicals are dismayed, the vote confirming what they say is their worst fear that the authority Scripture is no longer the rule of faith and practice.
The Bishops had said that the report was not the “final word” but rather a stepping stone toward greater inclusiveness.
But General Synod’s decision not to “take note” means that it can no longer play any part in future discussions. The slate is wiped clean; who gets to write the next chapter is at this stage unclear.
Opening the debate, the Bishop of Norwich, Graham James, said he regretted “any pain or anger” the report may have caused.
The Bishop of Willesden, the Right Reverend Pete Broadbent, apologised to members who felt alienated by the report.
“I do want to apologise to those members of Synod who found our report difficult, who didn’t recognise themselves in it, who had expected more from us than we actually delivered, for the tone of the report,” he said.
“On behalf of the House, and without being trite or trivial, I’m sorry.”
Before the debate, protesters gathered outside the Church of England headquarters and champions of LGBT rights sang hymns.
The group, organised by the Lesbian and Gay Christian Movement and Changing Attitude now known collectively as One Body One Faith, with the support of Out and Proud African LGBTI and rights campaigner Peter Tatchell, called for the rejection of the bishops’ report.
Mr Tatchell said the Synod’s vote was a “victory for love and equality”.
The Reverend Bertrand Olivier, who’s gay, told the BBC the Church needed to reflect modern society.
He said: “I was ordained as an openly gay candidate then and it’s been going backwards ever since at the same time as the nation has moved on and we now have legal same-sex marriage.”
But the Right Reverend Pete Broadbent had said campaigners may be asking too much.
He said: “Our role is to hold the Church together and say we can only go as far as the whole church can agree. Campaigners are actually wanting us to go further, more hurriedly, than we necessarily can.”
The Synod earlier unanimously passed a motion urging the government to bring forward proposals to reduce the amount gamblers can stake on on fixed-odds betting terminals from £100 to £2.
The Bishop of St Albans, Alan Smith, said the debate concerned a “very focused form of betting which has caused huge suffering”.