Mr Clarke told judges at their annual Mansion House Dinner in London that "no-one can prove cause and effect" for why crime fell in the 1990s.
His comments come after former Tory home secretary Michael Howard recently criticised him for attacking high imprisonment rates.
...In his speech on Tuesday night, Mr Clarke said: "There is and never has been, in my opinion, any direct correlation between spiralling growth in the prison population and a fall in crime.
"Crime has fallen in Britain throughout a period of both rising prison populations, and throughout the same period of economic growth, with strong employment levels and rising living standards.
"No-one can prove cause and effect. The crime rate fell but was this the consequence of the policies of my successors as home secretary or, dare I gently hint, mine as chancellor of the exchequer at the beginning of a period of growth and strong employment? We will never know."
The justice secretary, who favours rehabilitation and community sentences, said crime had fallen in Canada in the 1990s after the prison population was cut by 11% and that crime did not rise significantly in Finland when similar measures had been taken there.
... Dame Anne Owers' warning that prisons are now increasing brittle came in her valedictory lecture to the Prison Reform Trust on Tuesday night.
She said prisons "had become better places" but progress in rehabilitating offenders was slow because of the growing prison population.
She called upon ministers to "do things differently" in the "age of austerity" and invest in alternatives to prisons.
"We now have an inflated prison system in a shrinking state. It [is] crucial to invest in 'not prison' - both instead of and after prison," she said.
...A Ministry of Justice spokesman said as part of the "rehabilitation revolution", the government was working towards providing a "fit for purpose prison estate" by building new prisons and closing "inefficient and worn out places".
"The government has announced its intention to conduct a full assessment of sentencing policy to ensure that it is effective in deterring crime, protecting the public, punishing offenders and cutting re-offending," he added.
Read the full article.
See the government's structural reform plan that includes an evaluation of sentencing policies including restorative justice:
Reform of sentencing and penalties
Ensure that the justice system protects the public and reduces reoffending by introducing more effective sentencing policies and considering the use of restorative justice for adult and youth crimes.