Darwin's first reflections on human progress were prompted by his experiences in the slave-owning colony of Brazil, and by his encounters with the Yhagan peoples of Tierra del Fuego. Harsh conditions, privation, poor climate, bondage and servitude, could impede human progress or cause degeneration. In the "Fuegians", Darwin thought he had witnessed man in his most "primitive wildness" (letter to Henslow, 11 April 1833). They represented both the yawning gap between wild and domesticated humans, and the unsettling proximity of the savage and the civilized. The Beagle carried three Yahgans who had been taken from their homeland by Robert FitzRoy several years earlier as part of a missionary enterprise. Darwin was struck by the progress that had been achieved through their forced migration to England: "in contradiction of what has often been stated, three years has been sufficient to change savages, into, as far as habits go, complete & voluntary Europeans" (Beagle Diary, p. 143). But he was also shocked by their rapid reversion to the primitive state, once they had been returned to their native land.
After the voyage, Darwin began to question the inevitability of progress. In his private notebooks, he modeled evolution after a tree of life or coral that was "irregularly branched" (Notebooks, B21), rather than linear, and with numerous dead ends. Fitness was relative to the conditions of existence which sometimes favored simpler forms or more instinctive behaviors: "absurd to talk of one animal being higher than another" (Notebooks, B74). The tendency toward increased complexity and variety, he suggested, was a bi-product of the abundance of life; retrogression was also possible, and was evident in some animal classes.
After the publication of Origin of Species, many of Darwin's supporters continued to believe that descent was propelled by an inward force or directed by design, while others such as Charles Lyell and Alfred Russel Wallace placed limits on natural selection as applied to human development. Another major point of controversy was the origins and unity of the human species, with researchers in ethnography and physical anthropology divided over theories of 'monogenism' and 'polygenism' and related questions about the supremacy of 'white races', the expansion of European empires, and the extinction or extermination of other peoples and cultures.
When Darwin wrote about the human races and the evolution of the 'higher' intellectual and moral faculties in Descent of Man, he drew on recent anthropology, comparative anatomy and zoology, surveys of 'primitive culture' and philology, as well as his own observations on human and animal behavior accumulated over three decades. Darwin argued forcefully for the unity of the human species, remarking that the dispute between monogenists and polygenists will "die a silent and unobserved death" when evolution is generally accepted (Descent 1: 235). He also employed the prevailing concept of civilization with its implied ranking of peoples, past and present, regarding their political, material, and technological advances as manifestations of their intellectual and moral powers. The "grade of civilization", he wrote, "seems a most important element in the success of nations" (Descent 1: 239). The implications of Darwinian theory for progressive, racial, and racist theories of human nature would remain one of the most controversial subjects of debate through the end of the nineteenth century and beyond.
Darwin’s first observations of the peoples of Tierra del Fuego were sent to his mentor, the professor of botany at Cambridge, John Stevens Henslow.
Letter 204: Darwin to Henslow, J. S., 11 April 1833
"The Fuegians are in a more miserable state of barbarism, than I had expected ever to have seen a human being.— In this inclement country, they are absolutely naked, & their temporary houses are like what children make in summer, with boughs of trees.— I do not think any spectacle can be more interesting, than the first sight of Man in his primitive wildness."
Charles wrote to his sister, Emily Catherine Darwin, about witnessing slavery in the Portuguese colonies, and expressed excitement at the prospect of England abolishing slavery in its territories. Slavery was outlawed in most of the British empire by an act of Parliament in August 1833 which took effect in the following year.
Letter 206: Darwin to Darwin, E. C., 22 May [– 14 July] 1833
"I have watched how steadily the general feeling, as shown at elections, has been rising against Slavery.— What a proud thing for England, if she is the first Europæan nation which utterly abolishes it.— I was told before leaving England, that after living in Slave countries: all my opinions would be altered; the only alteration I am aware of is forming a much higher estimate of the Negros character.— it is impossible to see a negro & not feel kindly towards him".
Many supporters of Darwinian theory continued to believe in the primordial separation of the human races, and pursued researches on the fixity of racial types. The Anglican clergymen and educator Frederic Farrar wrote several articles in support of the polygenist theory of human descent.
Letter 4933: Farrar, F. W. to Darwin, 6 November 1865
"so far as I can see, History, & even Tradition, as far back as their primeval dawn, prove to us the existence of the several human races unchanged from their present physical characteristics; & if it be demonstrable that, under every possible variety of external influence & physical condition, the chief existing races have remained unaltered for say 5000 years—is not this a very strong argument for the Polygenist?"
Darwin asked the English settler James Philip Mansel Weale to distribute his questionnaire on expression in the Cape Colony, and received a set of replies from the South African native, Christian Gaika. Darwin was impressed by Gaika's knowledge of English and used some of his observations in Expression. Weale, however, was skeptical about the state of civilization of the natives.
Letter 5617, Darwin to Weale, J. P. M., 27 August 
"You have been extremely kind in taking such great trouble about expression, which is a subject that interests me to an unreasonable degree. That I shdreceive answers written by the brother of a Kaffir chief is a truly wonderful fact in the progress of civilization"
Letter 5722, Weale, J. P. M. to Darwin, [10 December 1867]
"You speak sanguinely about the civilization of the natives, & the fact that Christian Gaika can write. This appears to me an error into which most people in England fall... Although by no means desirous of running down Missionary work, I must own that in my opinion their teaching is to little effect.... The principles on which they work ... is almost purely an appeal to the emotions, & the longer a Kafir has been on a Mission Station the worse servant he is."
Just prior to the publication of Origin of Species, Darwin discussed his views on progress in a letter to Charles Lyell, insisting that there was no inherent tendency toward complexity of structure. This remained a point of dispute between many of Darwin’s scientific supporters, including Lyell, the Italian botanist Federico Delpino and the American zoologist Alpheus Hyatt. In the last edition of Origin (1872), Darwin tried to clarify his position: "natural selection, or the survival of the fittest, does not necessarily include progressive development----it only takes advantage of such variations as arise and are beneficial to each creature under its complex relations of life" (Origin, 6th ed, p. 98).
Letter 2503: Darwin, C. R. to Lyell, C., 11 October 
"the theory of natural selection ...implies no necessary tendency to progression. A monad, if no deviation in its structure profitable to it under its excessively simple conditions of life occurred, might remain unaltered from long before Silurian age to present day. I grant there will generally be a tendency to advance in complexity of organisation; though in beings fitted for very simple conditions it would be slight & slow. How could a complex organisation profit a monad? if it did not profit it, there would be no advance.— "
Letter 6728: from Charles Lyell, 5 May 1869
"I feel that Progressive Development or Evolution cannot be entirely explained by Natural Selection I rather hail Wallace’s suggestion that there may be a Supreme Will & Power which may not abdicate its functions of interference but may guide the forces & laws of Nature."
Letter 6866: From Federico Delpino, 22 August 1869
"Perhaps because of intellectual error but certainly with profound conviction, I am a teleologist. I believe that the rational principle by which man is conscious of being animated rules all other beings as well, plants included. And I believe that the first cause of all variations, whether for better or worse, progressive or regressive, lies exactly in this inner principle, inborn in all things."
Letter 8658: to Alpheus Hyatt, 4 December 
"I have never been so foolish as to imagine that I have succeeded in doing more than to lay down some of the broad outlines of the origin of species. After long reflection I cannot avoid the conviction that no innate tendency to progressive development exists, as is now held by so many able naturalists, & perhaps by yourself. It is curious how seldom writers define what they mean by progressive development; but this is a point which I have briefly discussed in the Origin."
Darwin discussed the role of civilization in the struggle for existence between human races with the geologist Charles Lyell, the liberal Anglican clergymen Charles Kingsley, the naturalist Alfred Russel Wallace, and the philosopher William Graham.
Letter 2503: Darwin, C. R. to Lyell, C., 11 October 
I suppose that you do not doubt that the intellectual powers are as important for the welfare of each being, as corporeal structure: if so, I can see no difficulty in the most intellectual individuals of a species being continually selected; & the intellect of the new species thus improved, aided probably by effects of inherited mental exercise. I look at this process as now going on with the races of man; the less intellectual races being exterminated."
Letter 3439: Darwin to Kingsley, Charles, 6 February 
"It is very true what you say about the higher races of men, when high enough, replacing & clearing off the lower races. In 500 years how the Anglo-saxon race will have spread & exterminated whole nations; & in consequence how much the Human race, viewed as a unit, will have risen in rank."
Letter 4510: Darwin to Wallace, A. R., 28 [May 1864]
"Now for your Man paper, about which I shd. like to write more than I can. The great leading idea is quite new to me, viz that during late ages the mind will have been modified more than the body; yet I had got as far as to see with you that the struggle between the races of man depended entirely on intellectual & moral qualities.
Letter 13230: Darwin to Graham, William, 3 July 1881
"I could show fight on natural selection having done and doing more for the progress of civilisation than you seem inclined to admit. Remember what risks the nations of Europe ran, not so many centuries ago of being overwhelmed by the Turks, and how ridiculous such an idea now is. The more civilised so-called Caucasian races have beaten the Turkish hollow in the struggle for existence. Looking to the world at no very distant date, what an endless number of the lower races will have been eliminated by the higher civilised races throughout the world."
Charles Darwin, Notebooks, B 18-29; E 95-7 [available at Darwinonline]
John Lubbock, Pre-Historic Times (1865) [available at archive.org]
T. H. Huxley, "Methods and Results of Ethnology" (1865) [available at archive.org]
Carl Vogt, Lectures on Man (1864) [available at archive.org]
Alfred Russel Wallace, "The origin of human races and the antiquity of man deduced from the theory of natural selection", Anthropological Review 2 (1864): clviii-clxx [available at archive.org]
Adrian Desmond and James Moore, Darwin's Sacred Cause. London: Allen Lane, 2009.
Mary Cowling, The Artists as Anthropologist. Cambridge University Press 1989.
George Stocking, George. Race, Culture, and Evolution: Essays in the History of Anthropology. New York: The Free Press, 1968.
Robert J. C. Young, Colonial Desire: Hybridity in Theory, Culture, and Race. London: Routledge 1995.