Deputy Attorney General James M. Cole, the second-highest ranking official in the Justice Department and a veteran of U.S. law enforcement, announced Thursday that he is stepping down and will take a job in the private sector.

Cole’s departure means that the entire leadership of the Justice Department will have changed by early next year, when Attorney General Eric H. Holder Jr. is expected to leave. Tony West, the third-highest ranking Justice official, left last month to become general counsel of PepsiCo.

Possible successors to Cole include Loretta E. Lynch, the U.S. attorney for the Eastern District of New York, and Sally Quillian Yates, the U.S. attorney for the Northern District of Georgia and the vice chairwoman of Holder’s advisory committee, officials said.

As deputy attorney general, Cole ran the Justice Department’s day-to-day affairs. Although often operating behind the scenes, he was among the administration’s chief public defenders of the government’s bulk collection of Americans’ telephone records, a practice disclosed by former National Security Agency contractor Edward Snowden. He was also instrumental in formulating the department’s policy on marijuana in response to the legalization of the drug in Colorado and Washington state.

In an interview Thursday, Cole acknowledged the difficulty of balancing security priorities with civil liberties.

“If you just want to keep people safe and you’re willing to sacrifice people’s constitutional rights and their civil liberties, that’s not so hard,” he said.

“If you just want to protect people’s constitutional rights and their civil liberties and you’re willing to sacrifice their safety, that’s not so hard either,” Cole said. “The hard part is to do them both.”

Cole, who served as deputy attorney general for nearly four years, said he is most proud of his work spearheading initiatives that transformed the department’s approaches to clemency, drug enforcement policy, the recording of federal law enforcement interviews and the protection of the right to counsel.

Cole said he assessed the Bureau of Prisons’ re-entry programs and ordered the bureau to revamp its programs and eliminate those that were not proven to reduce recidivism. He also has worked closely with Holder on sentencing reform efforts.

“We have to stop filling our jails,” Cole said. “The only way you’re going to do that is to take people who are addicted and start dealing with addictions, dealing with their mental health issues, instead of just throwing them in jail.”

But Cole was also the face of the Justice Department in the controversy that erupted when it was revealed that the department secretly acquired telephone records of Associated Press journalists during a leak investigation. Holder had recused himself from the probe, which was being supervised by Cole.

“Part of what you get with this job is the responsibility for making some very tough calls at times,” Cole said when asked about the case. “That was one of the very tough calls to have to make. But what was really important out of that was the dialogue we then entered into with the press. I like the press a lot. I think they keep us and everybody else honest.”

After the controversy — and another one involving Fox News reporter James Rosen — Holder put in place tighter controls over the ability of prosecutors to subpoena the phone and other records of journalists, and he and Cole began meeting with top newspaper and television editors.

Cole played a critical role in the department’s case against Credit Suisse, which helped wealthy Americans hide billions of dollars from U.S. tax collectors. The bank pleaded guilty, agreeing to pay $2.6 billion.

“I started that the day I came in here, and we truly have transformed the way Swiss banks are running and what our relationship is with them in a big way,” Cole said.

Cole plans to leave at the beginning of next year when his successor is in place. He said he does not yet know what his next job will be. But, pointing to a model of the famous Bluenose schooner in his corner office overlooking Constitution Avenue, he said he plans to spend more time sailing.