The Seagulls saw off Barclays Premier League opposition in the shape of a depleted Newcastle side on Saturday, and have also played Liverpool and Sunderland in the last couple of years.
They will now meet the winners of the replay between two more top-flight sides, who drew 2-2 on Sunday.
Poyet admitted before the draw he wanted to face of the smallest clubs in the competition at home, but only got half his wish.
"Both teams play good football and that will make it very difficult for us," he told the club website.
"Both teams have also been doing well in the Premier League lately and their game showed that.
"Either team will be very difficult for us to play against because of the way that they play. I'm not saying it will be impossible for us because anything can happen in the FA Cup, but it will be a very difficult game for us.
"It was a very even game (on Sunday). It was not the best first half but then Swansea made changes which worked for them, then it was all Arsenal and they looked to have won it before an equaliser. It was a proper cup tie.
"Nothing changes for me as the manager but it is a great draw for the club, great draw for the fans and great for the players.
"We will all wait and see what happens in the replay between the two sides now and then we'll go from there."