(TheWrap) -- Simon Cowell has his work cut out for him: The "X Factor" creator has said he wants his show to knock off "American Idol" as the biggest show on television. But so far it can't even top "Modern Family."
Earning 12.5 million viewers in its debut Wednesday, "X Factor" was the second-highest rated and only the fourth-most-watched show of the night. It had a 4.4/12 rating in the 18-49 demographic prized by advertisers -- well below the 6.0/15 for the season premiere of ABC's Emmy magnet "Modern Family," which boasted 14.3 million viewers.
"X Factor" also scored lower ratings than Fox's far less hyped "New Girl," which premiered Tuesday to a 4.8/12 and 10 million total viewers.
See also: 'X Factor': Watch Stacy Francis and Chris Rene's Performances (Video)
The "X Factor" ratings were solid, to be sure, but Cowell didn't do himself any favors by setting impossibly high standards for the show. He said at the Television Critics Association summer press tour that he wanted to top "Idol," quipping, "You don't enter into something for the silver medal."
"Idol," which Cowell left to launch "X Factor" in the U.S., averaged an 8.5/23 in the demo last season. "X Factor" scored half the viewers of its most recent premiere, in January, which had 26.1 million viewers. (That was the lowest-rated "Idol" premiere since it debuted in June 2002 to 9.9 million total viewers -- a number that "X Factor" easily topped.) Fox was the highest-rated network overall Wednesday, thanks to the consistent "X Factor" ratings during the premiere from 8-10 p.m.
Also read: 'Modern Family,' 'Community' Creators Discuss Twitter Feud, Primetime
ABC was second. "The Middle," airing at 8 p.m., had a 3.0/9 and 9.6 million total viewers. "Family" aired its hourlong episode at 9. The new drama "Revenge," premiering at 10, had a respectable 3.4/9 and 10.1 million total viewers.
CBS had a 3.1/9 and 10.4 million total viewers for "Survivor: South Pacific" had 8. The season premiere of "Criminal Minds" at 9 had a 4.1/10 and 14.1 million total viewers, making it the night's second-most-watched and third-highest-rated show. Ted Danson's debut on the season premiere of "CSI" at 10 had a 3.1/8 and 12.6 million total viewers -- enough to top "X Factor" in total viewers and be the night's third-most-watched show.
There was little good news for NBC. "Up All Night" at 8 scored a 2.3/7 and 6 million total viewers. As expected, it was down significantly from its strong debut last Wednesday at 10 -- a much more forgiving timeslot. "Free Agents" at 8:30 had a soft 1.3/4 and 3.9 million total viewers -- also down from its debut after "Up All Night" last week. The premiere of "Harry's Law" had a 1.2/3 and 7.3 million total viewers. The premiere of "Law & Order: SVU" at 10 had a 2.3/6 and 7.6 million total viewers.
On the CW, the second episode of "H8R" at 8 had a .4/1 and 1.1 million viewers. "America's Top Model" at 9 had a .7/2 and 1.6 million total viewers.