Start with why it is not silly. Everything you could cite as evidence for a world beyond experience — the solidity of a table, the testimony of others, the results of an experiment — is itself an experience. You have never encountered matter as such; you have encountered appearances, and inferred matter behind them. The idealist asks what that inference is resting on. It is a serious question, and the answers are less obvious than they feel.
Berkeley gave the boldest version: esse est percipi, to be is to be perceived. He arrived at it partly as an empiricist. If all our ideas come from experience, then the idea of a material substance — something underlying qualities but never itself perceived — is an idea we could never have acquired. It is not merely unproven but unintelligible. What exists, then, are minds and their ideas.
The obvious objection is that things persist when nobody is looking. Berkeley's answer was that God perceives everything continuously, which secures the world's stability. Whatever one makes of that, note that he was not claiming the world is imaginary or subject to your will: the ideas we receive are involuntary, orderly and lawful, and the difference between perceiving a fire and imagining one is exactly as real for him as for anyone. He thought he was defending common sense against the sceptical consequences of positing an unobservable substance.
Kant is the more influential and more moderate case, and the label matters: he called his position transcendental idealism and pointedly distinguished it from Berkeley's. His claim is that space, time and the categories such as causation are not features we discover in a mind-independent world but the forms any experience must take for beings like us. So we have genuine, objective knowledge of the world as it appears, and no access whatever to things as they are in themselves. This explains how mathematics and physics can be both necessary and about experience — at the price of a permanent limit on what metaphysics can reach.
Hegel rejected Kant's unknowable thing-in-itself as incoherent: to say there is something we cannot know anything about is already to say something about it. His absolute idealism treats reality as the development of rational structure through history, with the opposition between mind and world something that thought works through rather than confronts permanently.
ObjectionsWhy most philosophers aren't idealists.
Simplicity. Realism explains the order and inter-subjectivity of experience straightforwardly: there is a world, and we perceive it. Idealism must add machinery — God, or the structure of mind — to do the same work.
The success of science. Physics describes a universe that existed for billions of years before any perceiver. An idealist can accommodate this, but the accommodation looks strained where realism needs none.
Unmotivated. The strongest response is that the argument from "we only ever encounter experiences" to "only experiences exist" is invalid. That perception is our access to the world does not make the world identical to perception — inferring from the medium of access to the nature of what is accessed is precisely the step that needs defending.