The pagespeed test is synthetic & many items (not all) that reduce your score have zero real-world impact on your site usability or ranking.
For example - you loose points for loading Google fonts direct from Google servers,.. even though this is how they're supposed to be used & is usually much faster than loading them locally. You also loose points for not caching Google analytics code - something that is designed not to be cached anyway!
There are some obvious things like image optimization that you should look at - but the pagespeed scoring system will deduct the same number of points if your could save 2kb or 20mb, so even this metric should be looked at objectively.
CSS, HTML & JS minification is worth doing - but again, some sites don't render properly when minified, so it depends on your specific setup.
I just did a bit of work on Shrek's site - it was originally scoring 23 on mobile and 28 on desktop.... last check gave results of 73 on mobile and 82 on desktop. Very little of what I did had any real world impact though - most of it was just to satisfy the synthetic scoring. I replaced 2 image files and saved 4.3kb (A tiny amount of data) and the score jumped up 10 points,... so the scoring is not accurate at all...