After slavery was abolished in New York, early in the nineteenth century, slaves who had been agricultural laborers tended to migrate to Manhattan and many came to the area around Collect Pond where black freemen had made their homes and by the time Collect Pond was drained and filled in, these freemen and freed slaves had come to be the most prevalent as well as longest-term residents. As impoverished immigrants began to arrive in increasing numbers, the African Americans whom they joined proved to be the most prosperous of the destitute masses in the area. Bear in mind, however, that this prosperity was entirely relative; African Americans weren't rich, they continued to be treated as second class citizens (or worse), and they were severely restricted in their means of earning money. They were, in this instance, simply not at the bottom of the economic ladder for a change.
An 1857 report by a commission investigating Manhattan's worst housing conditions gives an explanation. By that time, as a result of the potato blight, Irish immigration had increased dramatically and the Irish had become the largest of the pauper classes. From the report:
But we must pass over without description hundreds of dilapidated, dirty and densely populated old structures which the committee inspected in different wards. In most of them the Irish are predominant as occupants, though in some streets negroes are found swarming from cellar to garret of tottering tenant houses. In this connection, it may be well to remark, that in some of the better class of houses built for tenantry, negroes have been preferred as occupants to Irish or German poor; the incentive of possessing comparatively decent quarters appearing to inspire the colored residents with more desire for personal cleanliness and regard for property than is impressed upon the Whites of their own condition. In one building the committee remarked much neatness and seeming comfort, with not a little evidence of a prolific habit on the part of families which might not have pleased a Malthusian philosopher. In answer to inquiries, and in fact by ocular demonstration, it was ascertained that nearly all the inhabitants were practical amalgamationists — black husbands and white (generally Irish) wives making up the heads of constantly increasing families.Some, perhaps most, of the African Americans who achieved a slight economic advantage over their immigrant neighbors did so by running grocery-grog shops, taverns, oyster-bars, and — most notably — dance halls.
Charles Dickens was one of the first writers to call attention to this relative prosperity. In his write up a trip to the US in 1842, he noted a bit of African American prosperity in a Five Points dance hall. After describing the horrible conditions of life in the district (as many others were doing then and would continue to do), he descends into a place called Almack's and tells us: "Heyday! the landlady of Almack's thrives! A buxom fat mulatto woman, with sparkling eyes, whose head is daintily ornamented with a handkerchief of many colours. Nor is the landlord much behind her in his finery, being attired in a smart blue jacket, like a ship's steward, with a thick gold ring upon his little finger, and round his neck a gleaming, golden watch-guard. How glad he is to see us! What will we please to call for? A dance? It shall be done directly, sir: 'a regular break-down.'"
A decade later, another sensationalist writer asked his readers to "see the squalid females, sottish males and half-starved urchins, perching about the windows, stoops and cellar doors [of Five Points], like buzzards on dead trees, viewing the dead carcass beneath." He tells us "The population of the Points is about equally divided between whites and blacks. The blacks however are, for the most part, the rulers; they own and keep a majority of the drinking and dance-houses."
Like Dickens, this moralist describes the best-known of the Five Points dance-houses: Almack's, also known as Pete Williams'. There were other dance halls in Five Points and many other establishments and tenements owned and leased by African Americans, but Pete Williams' was the one successful business that upper class white men patronized and it's the one that continued to show up in the press. This writer says,
But now it is time for Pete Williams's ball to open. Let us go down. We enter a cellar, where we see a few males and females, black, yellow, and white, seated or swaggering about the room; as many males smoking segars and swearing off some story of the day. Upon a sort of platform sit two or three negroes representing the orchestra, and opposite is the bar, behind which stands a negro, or Pete (who is a negro) himself, dealing out whisky, tobacco, beer, and segars, at three cents a glass, or a penny apiece. The music commences, and out sally two or three cotillions of this piebald party, and away they whirl in a most disgusting and revolting manner. The negroes seem to attract the most attention. What a commentary upon the authorities of the city. This is certainly the most — &msdash; [in the original text, two words are omitted as if censored]. Bah! let us get out, my senses refuse to behold longer such scenes. For two or three hundred yards square there is not a house or person worthy of respect; but all seem to partake of the polluting atmosphere which floats about, freely.Pete Williams died the same year that this diatribe was printed and the New York Times gave him a short tribute, saying for many years he ran "the dance-house of the Five Points" which was a "renowned place of resort," popular with men of importance, including Governors, members of Legislatures from various States, and prominent men from other countries." The obit writer apparently couldn't resist closing the piece with a nonchalant bit of contemporary racism, saying he hoped that "Pete has 'gone where the good darkies go.'"
That same year some Methodist ladies set up a mission in Five Points, aiming to bring some evangelical truth to the district's Roman Catholics, Jews, and heathens. An historical account of their work gives results of a some shoe-leather research in one block, profiling the its inhabitants' mix of nationalities, races, religions, and levels of illiteracy and including a piece of information that helps us understand one aspect of the economy of the place: "In the same block there were 33 undergound lodging-houses, ten feet below the sidewalk, and 20 of the vilest grog-shops in the city. During five hours on the Sabbath, two of these grogshops were visited by 1054 persons, — 450 men and 445 women, 91 boys and 68 girls."
About the time this account appeared, the New York Clipper, the city's entertainment paper, printed a brief memoir which once again described a visit to Pete Williams' and gave an indication of its profitability. The author tells us "Upon certain evenings of the week it was a carnival of fun to step down from Centre street to Pete Williams' when the lamps were ablaze. ... There was 'life' in Pete Williams', I tell you, and gay life, too." Calling it the "Hotel de Williams" and a "celebrated shantee," he says "the highest as well as the lowest class of people in the city of New-York visited Pete Williams." Like Dickens before him, he gives us evidence of prosperity: "Pete had as handsome a Creole to receive them, and as good a bottle of wine to crack for them, as any other man on this Island of Manhattan, 'and more too.'" He says, "Pete's girls were not only the handsomest specimens of their color in the United States, but the most agile and beautiful dancers I've ever seen. ... When the quadrille was finished, [the famous performers] Johnny Diamond or Juba gave us a jig, and let me tell you that 'a shower of browns' was not the reward of a fine specimen of heel-and-toe-ology in that house, but good bright solid half-dollar pieces and bills on the 'Chemical Bank,' in a match dance."
 New York African Burial Ground Archaeology Final Report (pdf)
February 2006 CHAPTER 2. DOCUMENTARY EVIDENCE ON THE ORIGIN AND USE OF THE AFRICAN BURIAL GROUND
 Gotham: a history of New York City to 1898, The History of NYC Series, by Edwin G. Burrows and Mike Wallace (Oxford University Press US, 2000) and Free Blacks and Slaves In Dutch Manhattan African American History of New York Before the British, Nov 19, 2009, by Melissa Cooper.
 Report of the Select Committee appointed to examine into the condition of Tenant Houses in New-York and Brooklyn, Documents of the Assembly of the State of New York, Volume 3, 1857.
 American Notes by Charles Dickens (Harper & Brothers, 1842) Here's the relevant bit of text:
Let us go on again and ... plunge into the Five Points. But it is needful, first, that we take as our escort these two heads of the police, whom you would know for sharp and well-trained officers if you met them in the Great Desert. ... What place is this, to which the squalid street conducts us? A kind of square of leprous houses, some of which are attainable only by crazy wooden stairs without. What lies beyond this tottering flight of steps, that creak beneath our tread? — A miserable room, lighted by one dim candle, and destitute of all comfort, save that which may be hidden in a wretched bed. ... Glimpses of New York city; Glimpses of New York city; by William M. Bobo (J. J. McCarter, 1852)
Ascend these pitch-dark stairs, heedful of a false footing on the trembling boards, and grope your way with me into this wolfish den, where neither ray of light nor breath of air appears to come. ...
Mount up these other stairs with no less caution (there are traps and pitfalls here for those who are not so well escorted as ourselves) into the housetop; where the bare beams and rafters meet overhead, and calm night looks down through the crevices in the roof. ... From every corner, as you glance about you in these dark retreats, some figure crawls half awakened, as if the judgment hour were near at hand, and every obscene grave were giving up its dead. Where dogs would howl to lie, women, and men, and boys slink off to sleep, forcing the dislodged rats to move away in quest of better lodgings.
Here, too, are lanes and alleys, paved with mud knee deep: underground chambers, where they dance and game; the walls bedecked with rough designs of ships, and forts, and flags, and American Eagles out of number: ruined houses, open to the street, whence, through wide gaps in the walls, other ruins loom upon the eye, as though the world of vice and misery had nothing else to show: hideous tenements which take their name from robbery and murder; all that is loathsome, drooping, and decayed is here.
Our leader has his hand upon the latch of "Almack's," and calls to us from the bottom of the steps; for the assembly-room of the Five-Point fashionables is approached by a descent. Shall we go in? It is but a moment.
Heyday! the landlady of Almack's thrives! A buxom fat mulatto woman, with sparkling eyes, whose head is daintily ornamented with a handkerchief of many colours. Nor is the landlord much behind her in his finery, being attired in a smart blue jacket, like a ship's steward, with a thick gold ring upon his little finger, and round his neck a gleaming, golden watch-guard. How glad he is to see us! What will we please to call for? A dance? It shall be done directly, sir: "a regular break-down."
The corpulent black fiddler, and his friend who plays the tambourine, stamp upon the boarding of the small raised orchestra in which they sit, and play a lively measure. Five or six couple come upon the floor, marshalled by a lively young negro, who is the wit of the assembly, and the greatest dancer known. He never leaves off making queer faces, and is the delight of all the rest, who grin from ear to ear incessantly. Among the dancers are two young mulatto girls, with large, black, drooping eyes, and head-gear after the fashion of the hostess, who are as shy, or feign to be, as though they never danced before, and so look down before the visitors, that their partners can see nothing but the long fringed lashes.
But the dance commences. Every gentleman sets as long as he likes to the opposite lady, and the opposite lady to him, and all are so long about it that the sport begins to languish, when suddenly the lively hero dashes in to the rescue. Instantly the fiddler grins, and goes at it tooth and nail; there is new energy in the tambourine; new laughter in the dancers; new smiles in the landlady; new confidence in the landlord; new brightness in the very candles. Single shuffle, double shuffle, cut and cross-cut: snapping his fingers, rolling his eyes, turning in his knees, presenting the backs of his legs in front, spinning about on his toes and heels like nothing but the man's fingers on the tambourine; dancing with two left legs, two right legs, two wooden legs, two wire legs, two spring legs — all sorts of legs and no legs — what is this to him? And in what walk of life, or dance of life, does man ever get such stimulating applause as thunders about him, when, having danced his partner off her feet, and himself too, he finishes by leaping gloriously on the bar-counter, and calling for something to drink, with the chuckle of a million of counterfeit Jim Crows, in one inimitable sound?
 Death of a Noted Colored Man, New York Times, November 29, 1852, Wednesday, Page 8, 822 words
 Sunshine and shadow in New York by Matthew Hale Smith (J. B. Burr and company, 1869)
 Here's the brief reminiscence from the Clipper:
I probably can recite as many strange reminiscences of that classical region known as the "Five Points" as any man my age, and I think I can give a good many old residents odds of a few years, and beat them on those matters.
At the period of which I write I was merely a boy, or at least of that age which youngsters now-a-days consider to be manhood, viz., turning nineteen years. I was brought up in the "Bloody Sixth Ward," and my life was identified then, as it has been since, to a great degree, with its habits and institutions.
What boy ever lived or was reared in this great city had a better opportunity of seeing and knowing the difference between the good and the bad than one brought up in the Sixth Ward? And although it has borne a hard ... [illeg.] noon-tide of my life, willing to that that locality, even in a ... [illeg.] -sideration and thought, says, has always been "more sinned against than sinning."
At the time of which I speak, and for a few years after, there was sport there — aye, by the gods, and real sport, too. Upon certain evenings of the week it was a carnival of fun to step down from Centre street to Pete Williams' when the lamps were ablaze.
There was "life" in Pete Williams', I tell you, and gay life, too, sonny boy. The writer had returned from the Free Academy, had entered his name as a pupil in one of the best gymnasiums in the city, and being of a truant turn of mind, and fond of excitement, he desired to better relaxation than to step down for a few minutes in the evening and take an inventory of the ladies and gents visiting Hotel de Williams.
It is almost needless for me to speak to the readers of the Clipper of the wild and giddy sport that might be indulged in there, and the dangers that might have been passed if that sport was carried to extremes. But if any one had a license for his extravagance in that matter, it certainly was a "Sixth Ward boy."
There is a wonderful history to be written of that locality, and especially Pete Williams' house. That was the institution of the "bay," as it has been more recently called, and a number of those who since then have arose to the position of City Fathers, and other prominent places in our city government, have spent their wildest and most boisterous house in that celebrated "shantee." The highest as well as the lowest class of people in the city of New-York visited Pete Williams.
No one, I presume, who knows anything about the place, will attempt to deny this. As for my part, I am willing to make my "affidavy" that I have seen some of the biggest bugs in this city spread themselves in Little Water street. But Pete had as handsome a Creole to receive them, and as good a bottle of wine to crack for them, as any other man on this Island of Manhattan, "and more too."
Oh, if the walls of that rough and uncouth "crib" had audible language, what a shriek of dismay and misery they could send through some "brown stone fronts" that I know of now on Fifth Avenue.
So much for a boy's recollection. Those sports who used to revel in wine and Southern Creoles, and exhausted their mirthfulness and purses in the heart of the Five Points (hard as it may sound) in those days, have become staid and dignified citizens, and may be seen every afternoon promenading on the twenty-five cent side of Broadway.
It is a notorious fact, that Pete Williams received the best and most lucrative part of his patronage from those who were bound by connubial ties to home. But since those men were of a sportive disposition, there certainly was no place in this town where they could enjoy themselves better. This advantage they surely indulged in to their heart's content.
I declare, it makes my heart light to think of that house. Many and many a splendid gallopede and reel I've footed there. Pete's girls were not only the handsomest specimens of their color in the United States, but the most agile and beautiful dancers I've ever seen. It was a terpsichorean treat, I tell you, to see some of 'em "forward two." Then when the quadrille was finished, Johnny Diamond or Juba gave us a jig, and let me tell you that "a shower of browns" was not the reward of a fine specimen of heel-and-toe-ology in that house, but good bright solid half-dollar pieces and bills on the "Chemical Bank," in a match dance.
After Johnny beat Juba he became a little stiff and mazy, and the desire was so great to see him "throw himself," that we remember him often to have made his nice little twenty-five or fifty ... [illeg.] out of Williams' Hotel.
If Johnny Diamond had been wise and temperate, he would have been a millionaire to-day. But he was "as good as they made 'em" every time. He was gifted, generous, and fond of his mother. I know this, because I grew up with him; and no matter where he performed, if I couldn't be allowed behind the scenes, no Johnny Diamond would dance there.
But the Clipperites, who are generally "right smart fellows," and want to hold on to the thread of a narrative, will naturally inquire how about the "Hour with the Convict."
Well, I'll tell you. Just as I left Pete Williams' one night, and carried with me a few glasses of the ruby, I stopped into C. (excuse me, I don't like to mention proper names) Brown's mills, I mean. ... Thus perished one of the once most intelligent and respected, but subsequently one of the most depraved and obdurate convicts that had ever cursed the State Prison with his presence.
The night alluded to was a night of terror to me. I was comparatively young and unsophisticated, although I was fond of sport, and would be, if possible, where it was going on. I got more than I wanted on the occasion referred to. It took the starch out of my collar and made such an indelible impression upon my mind as to make mere member as long as I live, that never-to-be-forgotten "hour with a convict."
-- An Hour With a Convict. A Reminiscence of the Five Points. Written expressly for the New York Clipper by John F. Gilwer. New York Clipper Annual. 1855-66.