As he launched his attack on a Home Office beset by scandal, the shadow Home Secretary scented political blood. 'These figures reveal the extent of the crisis overwhelming the criminal justice system,' he said.' More crimes are being committed and more people are getting away with it. The vast majority of offenders are getting away with even the most serious offences ... We have to ensure more offenders are caught and convicted.'
The speaker wasn't the Conservative David Davis, laying into Labour's new incumbent John Reid. It was actually Jack Straw in February 1996, citing statistics drawn from a book that I had just published, In the Name of the Law. They showed that, across a range of the worst violent and sexual offences, recorded crime totals had soared, while the chances of being brought to justice had plummeted. Since the Conservative government's election in 1979, Straw said, robbers, burglars, rapists and other sex attackers and those responsible for the most brutal physical assaults had become steadily less likely to be convicted.
Davis, the present shadow Home Secretary, could quite easily make the same allegation now. Ten years after Tony Blair famously pledged that Labour in power would be 'tough on crime, tough on the causes of crime', an Observer investigation reveals that conviction rates - the percentage of recorded offences that result in a guilty finding in court - have dramatically decreased. Robbers, rapists, other sex offenders and attackers who inflict life-threatening injuries are committing many more crimes, and have become much more likely to get away with them than when Labour was elected in 1997. Of the categories Straw identified, only burglary has fallen. Its conviction rate has increased - by 0.5 per cent.
Just as in 1996, to set out this analysis flies in the face of official rhetoric. 'The government is turning the tide in the fight against crime,' a Home Office spokesman said in the Nineties to rebut Straw, adding that crime was showing 'the biggest percentage fall for 40 years'.
Fast forward to January 2006, when like many Home Secretaries before him, Reid's predecessor Charles Clarke claimed that the problem was not crime itself but the fear of it. 'Despite the fact that most crime categories are falling, fear of crime is still too high and public perception is often at odds with reality,' he said. His answer was to set up a working group to 'to look at the statistics and find out why people do not believe them'.
Of course, it all depends which statistics you read. Instead of the annual tables of crimes reported to and recorded by the police, the government likes to cite the British Crime Survey, a regular sweep based on opinion poll-style interviews, which seeks to extrapolate how commonly people fall victim to different types of crime.
According to a written Home Office statement to me last week, this is 'considered the more reliable measure of violent crime, as it covers crimes that are not reported to the police', and its figures are not affected by changes in police recording methods. If we choose to believe the BCS, 'long-term trends show substantial declines in levels of violent crimes', with an 11 per cent fall last year.
This claim conceals some awkward facts. First, almost half the 2.4 million 'violent incidents' that the BCS estimates happened in Britain last year involved no injuries at all, and most of the rest caused only very minor ones. Just 2 per cent required a visit to hospital. Second, even the BCS accepts there have been 'statistically significant increases in stranger violence and in the proportion of offences receiving medical attention', and that when it comes to more serious assaults, police figures have always been much more accurate. The more serious the assault, the more reliable the police figures become: stabbings or beatings that leave their victim near to death have never been likely to pass unreported.
Pace Clarke, these are the crimes which arouse the deepest public concern, and they have been rising rapidly. In 1980, the first full year of the last Tory government, police in England and Wales recorded 4,390 'woundings or other acts endangering life'. In 1997, the year of Blair's first victory, the figure was 12,531. In 2004-5 it reached 19,425, nearly four times the level 20 years earlier. Convictions rose, but they did not keep pace. In 1980, 1,277 people were found guilty of life-threatening attacks, 29 per cent of the recorded total. In 1997, 1,864 were convicted - 14.8 per cent of perpetrators. In 2004-5 the number convicted was just 1,897, despite the increase in crimes - a rate of only 9.7 per cent.
In other words, the chances of being convicted for a really violent assault are only one third of what they were in 1980 - something that may well help to explain why such crimes have become more widespread.
Robbery - a category that runs from mugging to bank raids - presents a similar picture. In 1980, the police recorded 15,800 crimes, with 3,600 convictions and cautions - a rate of 22.7 per cent. By 1997, offences were running at 63,072, with 6,426 convictions and cautions - 10.2 per cent. Under Labour, Blair has personally driven police 'street crime initiatives' to get the robbery figures down, and from a peak of more than 100,000 in 2002, recorded robberies fell in 2004-5 to 88,710. Yet the rate of convictions and cautions fell too, with the 7,932 total representing 8.9 per cent of offences.
Sex crime presents a still more dismal picture. The recorded figures for rapes against females in 1980, 1997 and 2004-5 were respectively 1,200, 6,281 and 12,867 - an almost elevenfold increase in 25 years. The corresponding conviction totals and percentage rates were 457, or 38 per cent; 576, or 9.2 per cent, and 704, just 5.5 per cent: the chance that the perpetrator of a recorded rape would ever be convicted was one seventh as great in 2005 as it was in 1980.
For other sexual assaults on women, recorded offences rose from 13,340 in 1987 (the 1980 figure is not comparable), to 18,674 in 1997, and to 26,709 in 2004. Here the total of convictions plus cautions actually fell - from 3,529 to 3,401 to 2,951, a decline in the conviction rate over just 18 years from 26.5 per cent to 11 per cent.
So to burglary, a relative success story. In 1980, police recorded 618,390 offences. This figure rose to 1,015,175 in 1997, but by 2004-5 it had fallen back to 679,973. In 1980, there were 65,700 convictions, a rate of 10.6 per cent. By 1997, they had slumped to 31,703, just 3.1 per cent. In 2004, there were still fewer convictions (24,252 in all), but the rate had slightly improved - to 3.6 per cent. One of the reasons for the fall is that when they are convicted, burglars have become far more likely to go to prison. Other kinds of theft, such as shoplifting, have meanwhile increased.
Whatever the Home Office says, across the board, the picture of Britain and violent crime is not a comfortable one.
Conviction rates fell under Tories and Labour alike, but the intense politicisation of criminal justice means that the first difficulty in seeking to uncover what these figures tell us is merely to establish that some of the rise in serious violent and sexual crime is real, as opposed to a consequence of increased reporting or more accurate recording.
In interviews with judges, prosecutors, Home Office spokespeople and senior police officers over the past month, I often encountered what sounded like an institutionalised reluctance to accept this. 'Any rise in reported rape is a good thing, in our view,' said the Met's Deputy Assistant Commissioner, John Yates. 'I think it's happening because our approach has changed significantly.'
Viv Goddard, a Crown Prosecution Service lawyer who has been prosecuting sex offenders for 15 years, agreed: 'My own view is that women know much more about what is available, and that they therefore have the confidence to say, "this happened to me."'
They could have been speaking in the mid-Nineties, when I was researching my original book. Then, practitioners explained the rise in reported rape and sexual assaults by pointing to the impact of a 1982 BBC documentary that depicted a detective cruelly interrogating a rape victim, which led to a transformation in the way the police dealt with such crimes. Many forces have continued to make improvements, as Yates said, working closely with groups such as Rape Crisis. But can such measures really account for an elevenfold rise in 25 years?
In its written statement, the Home Office claimed that similar factors explained the rise in violence - such as better police recording and more 'proactive' policing. They may be right, because what no one has is hard data: life-threatening assaults are not sufficiently numerous to make the BCS much help in determining whether their true level has increased, and until very recently, it did not study sex crime. But among seasoned practitioners close to the top of the criminal justice system, I found a marked uneasiness at the official state of denial, and concern that the figures truly reveal a rising incidence of personal crime.
Among the most forthright were senior police officers. 'Of course it's real,' said Terry Grange, Chief Constable of Dyfed-Powys in Wales and the Association of Chief Police Officers (Acpo) spokesman on 'private crime', which includes domestic violence and sex offences. 'As far as wounding is concerned, there's a much greater willingness to use weapons, and a much greater willingness to use violence all round.'
The ages of both victims and perpetrators was falling, he said, and there was evidence of 'an absence of restraint, especially among young people'. As for rape, 'a good part of the increase is also real,' he said. 'It's just foolish to pretend that this stuff was always happening, but it's only now that it's coming out. It's true of domestic violence, but it isn't true of rape. I do not believe that the level of attacks that we see now is purely the result of increased reporting, even if that does form part of the story.'
Grange identified an obvious cause: the surge in binge drinking, suggesting it exposed both male and female victims to much higher risk and disinhibited perpetrators. Matt Baggott, Leicestershire's chief constable, who is co-ordinating Acpo research on this subject, agreed. It was true, he said, that crime recording was more accurate, that victims were more ready to come forward and that there was a 'higher degree of intolerance' of both violence and sex crime in society at large. Nevertheless, he believed some of the increase was real, and that changes in youth culture and alcohol consumption were the underlying cause.
'If you stack up all the research on alcohol and young people, you'll find plenty of material on health issues and depression. One of the things the police pick up are assaults and offending behaviour. There needs to be a much bigger debate around the portrayal and use of alcohol and how it affects youth lifestyles; and to what extent profit is being put before people's wellbeing, of their chances of falling victim to ferocious assaults or serious sexual offences.'
All but one of seven judges I interviewed - who spoke on condition of anonymity - shared this analysis. 'My feeling is that people go further than they used to, and this is true of both sex and violence,' said a member of the Court of Appeal whose earlier career was spent at the criminal bar. 'Rapes used generally to involve simple vaginal penetration. Now, at least in stranger rape cases, usually there are aggravating factors - forced oral sex, buggery, manual penetration of the anus. I have a feeling that more humiliating and degrading experiences are being visited on the women. Violence cases are much nastier. Perpetrators have drawn and used knives more often. If there's a mob, they've all kicked the victim's head.'
Like the chief constables, a female judge at one of the country's busiest crown courts blamed changing social mores and alcohol. 'In the Eighties, the pubs shut at 11pm and the clubs by 1am or 2am. There were no alcopops, and women didn't start the evening by knocking back a series of double-measure shorts. Now, with dance music, ecstasy, and much harder drinking, young people are accustomed to staying out all night. The behaviour of women has changed especially: they're much more sexually aggressive and aggressive generally; some are armed. We see people accused of committing serious violence that we never saw 20 years ago, and some of them are women. At the same time, they are being exposed to much greater risks.'
Although it denies that serious crime may be increasing, somewhat paradoxically, the government already has a solution - what Blair and Reid have called 'rebalancing' the criminal justice system, away from the rights of defendants, and towards those of victims' needs.
Underlying their approach are two assumptions. The first is that the main reason why conviction rates have fallen is the trial process, which is too easily manipulated by defence lawyers who bamboozle juries into producing the 'wrong' result. The second is that further legal change - by a government that has already allowed courts to accept hearsay evidence and tell juries defendants' previous convictions, and try people twice for the same offence - would go a long way to putting this right. But new data suggests that success may be much harder to achieve.
The blunt truth is that over the entire period 1992-2006, most not-guilty verdicts recorded in the Crown Courts where all serious cases are tried have not been reached by juries at all, but by judges. In fact, where juries do reach a verdict, acquittals have steadily declined from 45 per cent of contested cases in 1992-3 to 38 per cent in 2005-6. Judges' acquittals, on the other hand, have risen. They fall into two categories: ordered decisions, entered when a trial is due to start because the prosecution says it cannot proceed, or the much less common directed ones, when the judge stops the trial because of a legal problem. Last year, juries found 5,927 defendants not guilty, less than half as many as the 13,894 freed by judges. Juries convicted 12,099 people, while 60,252 pleaded guilty.
Judges sometimes find ordering an acquittal as frustrating as victims. 'I would guess that at least half of the time, I'm uneasy,' said one. But usually, they have no choice. 'When you order an acquittal, it's not as if this is your considered decision,' another crown court judge said. 'You can insist on being given reasons, and make the prosecution's life difficult. But you can't make the prosecution proceed, and usually there is no choice because they've already told the defence. Legally, you're stuffed.'
Published CPS figures do not break down these figures down, but they have now been supplied for the first time to The Observer. They reveal that judge-ordered acquittals are overwhelmingly concentrated in serious sex and violence cases, which also happen to be the crimes where defendants are most likely both to plead not guilty, and to walk free.
Much of the debate about the way the criminal justice system treats victims focuses on their experience in court, and the way they are cross-examined. The figures demonstrate that when defendants are acquitted, in most cases, this has not actually happened.
In all, the prosecution asked judges for acquittals in 4,280 violence cases in 2005-6. Most involved charges of inflicting grievous bodily harm or wounding with intent. In 1,700 of them, the victim either failed to turn up, refused to give evidence, or, in the view of the CPS, did 'not support the case'. Some of these may have been domestic violence victims who no longer wished to proceed. Others will have been threatened or intimidated. A further 562 cases were dropped because other witnesses, not the victim, did not show up, refused to testify, or were considered to have become 'unreliable'.
In the same year, 432 robbery prosecutions had to be dropped for the same reasons. Judges fear that one reason may be the spread of what one called 'witness nobbling'. 'Revenge is being sought much more often,' said another. 'Sometimes you can see what's happening very clearly. It's down to one word: gangs.'
The rape figures give rise to other concerns: 1,958 rape cases went to crown courts in 2005-6. Just under half led to convictions, but of the 995 not-guilty verdicts, only 315 were reached by juries. Rape trials have long been controversial. For example, the Home Office is about to publish research showing that victims are still being questioned about their sexual history by the defence, years after an Act designed to bring this to an end. But if most acquittals are being ordered by judges, it suggests that this is a relatively minor reason for the low conviction rate. Much more significant are value judgements by prosecution lawyers. They may be well founded, but they are usually made on the sole basis of police statements and confidential detectives' assessments of victims and other witnesses.
Last year, the CPS asked judges to acquit 99 rape defendants because they considered a witness was 'unreliable', 64 because the victim's evidence would not sustain the case, and 110 because of a 'conflict of evidence' which, some might say, is what trials exist to resolve. In 71 cases, the victim asked to withdraw.
What does all this mean? First, that any search for sweeping, legislative 'rebalancing' solutions is doomed to fail. Of course, the reasons why cases are dropped before trials - and thousands more fail before being committed to crown courts - are influenced by what prosecutors know happens in the adversarial jury contest. But the details of this process have been given little scrutiny.
I asked Viv Godard, the rape prosecutor, what would make her think a case was likely to succeed, and what might persuade her to drop it. 'All I can do is read everything and sit back and think, "if I were defending this, what would I do?" If it comes down to the victim's word against the defendant's, you're looking for that little extra something that makes you think, "yes, that really did happen". You try to decide whether this person will stand up to cross-examination. But at the end of the day, it's a judgment.'
Given the high acquittal rate, she admitted, it was quite possible that CPS prosecutors often got it wrong. And if violence cases often fail because of intimidation, this opens up a very different vista of witness protection and surveillance.
On the ground, apparently small changes are under way that may make a big difference. One day last week, I sat for a morning in a busy police station while one of a new breed of CPS prosecutors, who are now responsible for charging any serious crime in place of the police, held meetings with detectives. Like an American district attorney, he discussed cases' details with them, instructing the officers to get further forensic evidence and witness statements to make the potential prosecutions more watertight before any charges were laid.
Later, I visited a CPS 'witness care unit' in Reading, where staff are trying to transform the relationship between prosecutors and their witnesses. Where it used to be routine for witnesses, including victims, to hear nothing between making their statements and being summoned to testify, staff now keep in regular touch, informing them of progress. Where basic needs such as transport or childcare are required, they make the necessary arrangements. Pilot schemes suggest this can reduce abandoned trials by 26 per cent, and if witness intimidation is beginning to make itself apparent, the police will be told immediately. Incremental progress looks possible.
Meanwhile, a meaningful debate awaits a realistic appraisal of the true level of violent and sexual crime, of its deeper roots, and the true reasons that prosecutions fail.
Until that answer is found, shadow Home Secretary after shadow Home Secretary can point at the figures and the falling conviction rates and say, with no fear of contradiction, that violent crime is up and conviction rates are down.
· Rapes
Recorded rapes of females in 1997:
6,281
Rapes in 2004-05:
12,867
Conviction rate for rape in 1997:
9.2%
Conviction rate for rape in 2004-05:
5.5%
· Robberies
Robberies recorded in 1997:
63,072
Robberies in 2004-05, after a peak in 2002:
88,710
Convictions and police cautions for robbery 1997:
10.2%
Convictions and cautions for robbery 2004-05:
8.9%
· Woundings
Woundings or other acts endangering life in England and Wales in 1997, the year Labour came to power:
12,531
Similar offences in 2004-05:
19,425
Conviction rate 1997:
14.8%
Conviction rate 2004-05:
9.7%