"This is becoming a bit of a habit, I'd prefer us to break the deadlock a bit earlier," France coach Didier Deschamps told reporters afterward.
"Until the final whistle is blown there is still a chance to win the game, especially against teams who you have to wear down," he said.
"I wouldn't say we were lucky but all the players put all the ingredients into winning the game."
France began their bid for a first European Championship title in 16 years with a nervous 2-1 win over Romania last Friday.
Double danger: England wary of sharks and Wales
But despite creating a host of scoring chances against the Balkan minnows in a stirring second half, it took Atletico Madrid striker Griezmann to rescue a pale version of Les Bleus with a 90th-minute header at the Stade Velodrome.
Deschamps replaced under-performing midfielder Paul Pogba with Anthony Martial while Kingsley Coman started instead of Griezmann for France's first competitive game at the Stade Velodrome since their triumphant World Cup campaign in 1998.
It meant a change from a 4-3-3 to a 4-2-3-1, with Arsenal striker Olivier Giroud playing solo up front.
But after a frustrating first half of few real chances for the hosts, Deschamps was quickly reconsidering his options.
Euro 2016: Hamsik's brilliance helps Slovakia down Russia
Albania, missing captain and ex-Marseille player Lorik Cana through suspension following his sending-off in the tournament debutants' 1-0 defeat to Switzerland, deployed a defence-minded 4-5-1 that made their conservative tactics clear.
And France, despite showing plenty of promise throughout, struggled to come up with the antidote.
Payet, who hit a stunning winner for France last Friday, showed promise with a great delivery for Giroud but the striker's header was a yard over.
Albania midfielder Ermir Lenjani's then sent a drive just wide but the Balkan minnows would threaten again.
France's offside trap was non-existent when Napoli defender Elseid Hysaj broke free on the right to collect a long ball and deliver a cross that deflected off Martial as Armando Sadiku ran in towards goal.
It was a brief scare, and France responded with a string of mistakes that left Deschamps holding his head in his hands before the half hour.
Switzerland draw 1-1 with Romania at Euro 2016
As widely expected, Pogba replaced Martial for the start of the second half, inspiring the hosts to a far improved performance.
Seconds after kick-off a great delivery from Giroud was helped on by Blaise Matuidi, but Coman's header from six yards out bounced just wide.
But France were given another scare when Bacary Sagna miscued a dangerous delivery that came off the post, bounced off the head of Ledian Memushaj and went out for a goal kick.
This time, the scare was big enough to prompt a rousing French response.
Moments later, Payet met Giroud's nice lay-off to hit an inviting delivery with the outside of his right boot towards the back post where Pogba rushed in only to fire inches over.
When N'Golo Kante made a timely interception to help set up Coman on the left, Giroud timed his jump to perfection but his header glanced wide on 67 minutes.
Coman was replaced by Griezmann soon after, and moments later Giroud rose to meet Patrice Evra's delicately chipped delivery to fire his header off the post.
Deschamps replaced Giroud with former Marseille favourite Andre-Pierre Gignac with 13 minutes remaining, but as the hosts maintained their push for the opener it was Griezmann who broke the deadlock when he rose to meet Adil Rami's cross to beat Lazio 'keeper Etrit Berisha.
It broke Albania's resolve and six minutes later Payet claimed his second of the competition with a fine curling strike that beat Berisha at his far post.
"My team resisted right to the end, but we didn't defend right to the end," said Albanian coach Gianni De Biasi.
"Especially at the end of the first half and the start of the second we had the better chances but when you don't score goals you can't get results."
COMMENTS
Comments are moderated and generally will be posted if they are on-topic and not abusive.
For more information, please see our Comments FAQ