This is an even murkier issue than you think. And let me state right up front that I don’t actually have any answers.
First, in terms of year of publication. This is the same issue that I raised regarding Yellowcake. In that book the short stories were published in various publications over a period of years, but the book itself just came out in 2013. So does that make it eligible or not? I think yes, but I don’t know what YALSA and the committee will decide. It appears that it is their call. And clearly the same would apply for Bone.
Second, the self-published question. This particular policy was only added to the Printz criteria in 2013, largely because of an increasing problem with determining eligibility on these types of items. So you can’t really apply that policy to your question about Bone, because there was no prohibition on self-published items in 2003.
There’s the question, of course, about whether the committee would have had the Bone books brought to their attention, since they weren’t published by a mainstream publisher. There’s the question about whether they would really have considered them “books” at that time.
These eligibility matters appear to be the call of the committee chair, who decides in consultation with YALSA staff. So there are several possibilities for what happened with Bone in 2002/3:
1. The committee never even considered Bone;
2. The committee considered Bone but decided it was ineligible because of previous publication;
3. The committee considered Bone but decided it didn’t meet the threshold of literary excellence.
So, as I said up front, I don’t have an answer.
Even the current policy about self-published books has the possibility of becoming problematic in future years, as the face of publishing changes. As you pointed out, if the books came out today, would Cartoon Books be considered self-published or not? I don’t know. This is totally a gut feeling, but I think if Cartoon Books only ever published Jeff Smith’s own books, I would consider that self-publishing. If they published other authors, I would consider it a publishing house. We could well see more of that sort of publishing.