UEFA's decisions, yet to be confirmed by European football's ruling body, came as a fresh blow to the beleaguered Turkish league amid a court investigation into match-fixing allegations which overshadowed the whole of last season.
Besiktas were fined 200,000 euros due to delays in payments to players and staff, they said in a statement to the Istanbul Stock Exchange where shares in the club are traded.
UEFA scrutinised Besiktas' accounts in the second and third quarters of 2011 and ruled that they breached articles of the ruling body's financial fair play regulations.
Besiktas had qualified for the second qualifying round of the Europa League and Bursaspor for the pre-qualifying rounds.
Shares in Besiktas tumbled more than 9 per cent on the news to 4.29 lira.
No details were available on the punishments for Bursaspor and Gaziantepspor. Media reports said sixth-ranking Eskisehirspor could get into the Europa League instead of Besiktas.
Besiktas Chairman Fikret Orman and the club's lawyers attended a meeting with UEFA's disciplinary committee for a second time on Wednesday after the club was earlier warned about its financial status and high debt.
Orman said Besiktas would appeal to the Court of Arbitration for Sport (CAS) regarding the ban, but added he was not very optimistic about a ruling in favour of the 13-times Turkish champions.
"We will defend the club's rights until the end, and will appeal to CAS as the first step," Levent Erdogan, a lawyer and club board member told Turkey's Dogan news agency.
"I think it's a dramatic decision, and it will hit us pretty badly, but there's nothing we can do," he added.
"It's impossible for us to receive such a suspension, it would be a murder of justice," Bursaspor club chairman Ibrahim Yazici told the state-run Anatolia news agency and added they would definitely appeal the ruling.
The other major Istanbul clubs Galatasaray and Fenerbahce, Besiktas's arch-rivals, will compete in the Champions League after taking the top two places in the domestic league.