If true, it would be the only known painting of Theo, although Vincent made several sketches of his brother, who supported him financially and was his lifelong confidant and friend.
"People have often thought it was funny that there were no portraits of Theo, given that they were so close," said museum spokeswoman Linda Snoek.
She said the portrait was made in 1887 while the pair lived together in Paris — a lesser-known period of Van Gogh's life, since the bulk of information about Vincent is derived from letters he sent to Theo.
The painting has long been in storage, but went on display at the museum in Amsterdam Tuesday as part of an exhibition on new findings about the painter's time spent in Antwerp and Paris in 1885-1888.
Though the brothers resembled each other physically, scholars determined the painting represents Theo by a number of factors.
Head researcher Louis van Tilborgh compared two paintings from 1887 with similar-looking men in suits set against a blue background.
"They are two small, detailed portraits that when you see them you think: they belong together," Van Tilborgh said in an interview with Dutch state broadcaster NOS.
The portrait of Theo shows he had rounder ears than Vincent did. The other portrait shows Vincent with long, angular ears, consistent with other artists' paintings of Vincent. That's before he famously self-mutilated one of his ears in December 1888.
In addition, Theo's goatee is more yellow-brown than Vincent's dark red beard, and Theo has shaven cheeks, consistent with photographs of him from the same period, while Vincent painted himself sporting mutton-chop sideburns.
In other discoveries, the museum said the bird depicted in the 1887 painting previously known as "Wheatfield with a Lark" is actually a partridge.
Van Gogh, who suffered from depression and mental anguish much of his life, shot himself in a wheatfield in 1890 and died shortly thereafter.