The hybrid flow shop group scheduling problem (HFGSP) with the delivery time windows has been widely studied owing to its better flexibility and suitability for the current just-in-time production mode. However, there are several unresolved challenges in problem modeling and algorithmic design tailored for HFGSP. In our study, we place emphasis on the constraint of timeliness. Therefore, this paper first constructs a mixed integer linear programming model of HFGSP with sequence-dependent setup time and delivery time windows to minimize the total weighted earliness and tardiness (TWET). Then a penalty groups-assisted iterated greedy integrating idle time insertion ( $\mathrm{P}\mathrm{G}\mathrm{\_}\mathrm{I}\mathrm{G}\mathrm{\_}\mathrm{I}\mathrm{T}\mathrm{I}$) is proposed to solve the above problem. In the $\mathrm{P}\mathrm{G}\mathrm{\_}\mathrm{I}\mathrm{G}\mathrm{\_}\mathrm{I}\mathrm{T}\mathrm{I}$, a double decoding strategy is proposed based on the earliest available machine rule and the idle time insertion rule to calculate the TWET value. Subsequently, to reduce the amount of computation, a skip-based destruction and reconstruction strategy is designed, and a penalty groups-assisted local search is proposed to further improve the quality of the solution by disturbing the penalized groups, i.e., early and tardy groups. Finally, through comprehensive statistical experiments on 270 test instances, the results prove that the proposed algorithm is effective compared to four state-of-the-art algorithms.