In "The Doll's House," do the dolls that came with the house hold significance in that they do not fit or belong?

This is an interesting idea. However, I must be honest.
The fact that the dolls are only mentioned once in the entire short story and that they
are given a very cursory description suggests to me that they are not really important,
and that we need to be careful when reading in to them any symbolic meaning. Far more
attention is given to the lamp, for example, which clearly can be argued to be symbolic.
Let us consider the description of the dolls that we are
given:

The
father and mother dolls, who sprawled very stiff as though they had fainted in the
drawing room, and their two little children asleep upstairs, were really too big for the
doll's house. They didn't look as though they
belonged.

Now, the phrase
"They didn't look as if they belonged," could be used to suggest that the dolls could be
symbolic of the Kelvey's, as you suggest, because they definitely do not belong to their
surroundings either. However, this is much more likely to be a rather humorous
description of the doll's house as being rather cheap and not well-made. In spite of all
the detail, let us remember that there were "big lumps of congealed paint hanging along
the edge." Therefore, I think it is more likely that this is not a symbolic comment as
such, and just relates to the shoddy presentation of this doll's house, which
nevertheless holds such importance for the Burnell girls.