I don't care about numbers either. I think the perception of a low number being worth more, in the mind or in dollars, is a carry over from the old days of collectibles, when they didn't exactly come off a production line, or were more hands on made, perhaps having a limited run, not likely to be in the thousands, as they are now.
As things changed over time, items are being made as collectibles rather than being a limited run item, that may be of interest to collectors. Possibly, in the old days, lower numbered items were better representations, as later in a run, defects from an overused mould may have become more apparent.