Milton James grew up in rural South Australia, the son of a signalman who was proud of his working class heritage. Milton's working life began in the mining industry, followed by qualification and considerable experience in the agricultural and horticultural industry.

In 1978, he travelled to India to work as a volunteer on a remote community development project in the poorest villages in the poorest state of India. This experience stirred him into studying social work and he began developing his career in rural South Australia and Victoria, with a particular passion for working with disadvantaged rural and remote youths.

In 1996, having just completed further studies in social work and working in Canberra, a visiting Aboriginal friend and mentor, the late Molly Dyer from Horsham in Victoria, noting his despondent state of mind, said to him; "Milton, this is no place for you. You don't belong here. You belong on the frontier." No sooner were these words spoken, Milton and his family left to work in the Torres Strait.

Some years later, it occurred to Milton that he had overshot the real frontier - remote area social work where young Indigenous people are truly deprived and where conventional forms of intervention had never operated successfully.