I believe you could fish out the texel values with a timer-based attack even without vertex texture support: Just draw N screen-sized quads, where N is the number of texels in your secret texture, using a fragment shader that samples the same texel for every fragment and consumes time in proportion to the texel value (which Steve already showed how to do). If you have a very fast GPU and/or bad timers, just increase the size of the quad and/or draw it several times until you get measurable differences.
As a newcomer to web development, I keep wondering if these restrictions on image data access are genuinely necessary. They are a giant pain in the bottom for the ordinary developer, yet there are ways for attackers to go around them, so why bother?
Because we have to. Origin restrictions are not something we can elide just because we think they are not "necessary". Security is of prime importance to many people. The news that your browser didn't respect origin restrictions would cause some companies to block the use of your browser on their site. This is a fact we have to live with.