Lot of detail missing in terms of make/models involved, but it could be that you are plugging a dynamic mike into the 3.5 mm line level input.
Line level inputs look for 1 or 2 volts from the source (e.g., the output level of a CD player or tuner, etc.) and dynamic mikes don't put out anything close to that, which is why many microphones (and phono cartridges) need a pre-preamp to bring the output up to the needed line level.
Some gear does have the appropriate preamp built in (think an integrated amp with a phono input or a tape recorder with a true mike input) but that doesn't sound like the case in your situation.