A brown dwarf is larger than a planet, but not large enough to be considered a star. Stars undergo fusion in their cores, but a brown dwarf lacks the mass necessary to initiate fusion. At the same time, they are more like stars than planets, and can have at atmospheric temperature of 2000 K. So these objects occupy a middle ground between star and planet.
Historically, brown dwarfs have been difficult to observe. They are small and relatively cool, which makes them fairly dim in the range of visible light. However with the rise infrared astronomy and all-sky surveys these brown dwarfs are now fairly easy to detect. This is particularly true for brown dwarfs which are companions to other stars, since the methods used to discover planets would also easily detect brown dwarfs.
This has given rise to a bit of a mystery, known as the brown dwarf desert. You can see this in the figure here, which shows the distribution of stars by spectral type for stars within 32 light years of us. As you go from left to right, the mass of the stars go from large to small. You can see that with smaller mass comes greater numbers. There are much more red dwarf stars (M-class) than there are sun-like stars (G-class). But once you get to brown dwarf mass, the number drops off significantly. Given the way stars form, one would expect even more brown dwarfs than red dwarfs.
This particular chart is for stars within 32 light years. At that range, the drop off is not simply due to brown dwarfs being dimmer and harder to detect. We’ve done all-sky surveys in the infrared range, and if the brown dwarfs were there we would have observed them. This also agrees with searches for extrasolar planets. The methods we use to detect planets work best when the planets are more massive. This is why, for example, the planets we’ve found so far tend to be more Jupiter sized than Earth sized. If there were lots of brown dwarfs we would expect to have found them orbiting other stars. We don’t see that.
So for some reason there seems to be much fewer brown dwarfs in the universe that we might expect. Why this might be is still a bit of a mystery, but one possibility comes from the fact that larger planets form earlier than smaller planets in the accretion disk of a young star. Since brown dwarfs are larger than Jupiter-size planets, they would start to form very early in the accretion disk of a star, and the drag of the accretion disk would cause their orbits to decay. As a result they would fall into the young star. Smaller planets, forming later, wouldn’t have the same problem. So there would seem to be an upper constraint to the mass of planets. This agrees with statistical studies of exoplanets.
Of course that doesn’t explain why there aren’t more single brown dwarfs in our neighborhood. It might be that given the dynamics of a stellar nursery brown-dwarf mass objects don’t tend to form. Instead they tend to coalesce into larger stars early. We still aren’t sure.
Some things remain a mystery for now.