Diddy should serve the remainder of his 50-month sentence in a federal prison as close to the New York metropolitan area as possible … at least according to the judge who imposed the sentence.

According to new legal docs, obtained by TMZ, Judge Arun Subramanian stops short of naming a specific prison where Diddy should do hard time and instead says Diddy should stay close to home.

Besides geography, the judge says Diddy should be shipped off to a prison with “any available substance abuse program, including the Bureau of Prisons’ Residential Drug Abuse Program (RDAP), for which he qualifies.”

We broke the story … Diddy’s legal team says FCI Fort Dix in New Jersey is the best landing spot for Diddy because there he can “address drug abuse issues” and “maximize family visitation and rehabilitative efforts.”

FCI Fort Dix is about 77 miles away from the courtroom where Diddy’s trial was held.

Prosecutors took “no position” on where Diddy should go to prison … and ultimately, the Bureau of Prisons has the final say on where Diddy serves hard time.

Wherever he ends up, Diddy needs to watch his back … because his longtime attorney Mark Geragos previously told Harvey on the “2 Angry Men” podcast Diddy will be a target for other inmates, even in a low-security setting.

It may all be for naught … Diddy’s already asked Trump for a pardon.