Last week, same-sex marriage was legal in 19 states.

After the Supreme Court's decision on Monday not to review the rulings of several appeals courts, same-sex marriage may soon be legal in 35 states.

Ted Olson and Tony Perkins joined Chris Wallace on Fox News Sunday to discuss the many legal issues surrounding same-sex marriage and if it should be decided by courts or states.

Olson suggested that the Supreme Court Justices likely saw that many federal judges at the district court and appeals court levels had consistently ruled that same-sex marriages bans are unconstitutional.

"I think what they saw was an overwhelming trend in the same direction and felt that the federal courts were handling this in an appropriate and proper way and decided not to weigh in," Olson said.

Perkins, on the other hand, suggested that the Supreme Court is simply letting lesser courts do their decision-making for them, which he characterized as a "back alley type, Roe v. Wade judicial decision."

"This issue is not going away, despite what the court may say," Perkins stated.

Watch the full debate in the clip from Fox News Sunday above.