those rules were set up when UNWRA was established in 1949. It was one of several UN bodies established during that period, each of which dealt with a specific refugee population; all of these (other than UNWRA) have since been disbanded. A single body for dealing with refugees was set up in 1951 (the UNHCR) and its rules derive from the
Convention relating to the Status of Refugees.
About the first aspect you mentioned: When our lovely conservative govt locks up refugees who've fled from their home countries, children born in this country are not given Australian citizenship and are kept locked up as they're considered every bit a refugee as their parents are...
Well, I'm talking about the "global" definition, not what a particular government might use. In any event, children of refugees being refugees (unless they've resettled) isn't totally a negative thing. Otherwise, the children lack the support given to refugees, but presumably cannot return to their "home" country for the same reason their parents can't, or else must be seperated from them. Worse, under some conditions children born to refugees may wind up without any citizenship at all, and denying them refugee status means they're totally adrift. Extending that to all future generations, however, is IMO taking things too far.
About the second aspect - If Palestinians don't lose their refugee status when getting citizenship in another country, why do countries like Lebanon refuse to give them citizenship? I'm guessing discrimination is at the root of it...
I assume the other Arab countries don't give Palestinians citizenship for several reasons. First of all, for all their portraying themselves as champions of the Palestinians, they actually treat them like crap. Other than in Jordan - which is a Palestinian-majority country - they tend to be quite despised.
Second, leaving them as refugees, and the camps in squalor, makes them a more potent propoganda weapon against Israel. By not giving them citizenship, it's easier for the host country to avoid being asked why they're leaving the camps undeveloped (as I noted above, UNWRA support would create an economic disincentive to investing money in the camps even when they are citizens - as can be seen in Jordan - but this way they can formally disavow responsibility).
Third, in the specific case of Lebanon, it's also probably a mix of a desire not to upset Lebanon's ethnic balancing act plus acrimony from the civil war.
About the small towns - I saw a documentary recently about one refugee camp in Lebanon. Sure, you could call it a small town if yr talking about a small shanty-town with conditions that are pretty bad and where the inhabitants have no future....
That illustrates my point, actually. The refugee camps are basically permanent settlements. But their rtaining refugee status means the government has no incentive to develop them beyond "shanty-town" status.
As I noted above, the retention of refugee status by children is not necessarily a bad thing. But it needs to be kept in mind when making comparisons of the size of refugee populations, as the OP does; the Palestinians may currently be the largest refugee population in the world, but their "competitors" are counted by a different definition, under which most of the Palestinians would not count as refugees.