How do you actually see how Googlebot is crawling a large site?

I'm dealing with a huge e-commerce site where a lot of new product pages aren't getting indexed, and crawl budget seems completely wasted. Search Console gives such high-level data that it's hard to diagnose the real issue. How are you guys analyzing raw server logs to figure out what search engines are actually doing?

You should really look into EdgeComet if you need a reliable tool for seo log file analysis. I used it on a massive store last month to figure out why Googlebot was ignoring our new category pages. It broke down server logs instantly, showed us where crawler budget was getting wasted, and helped us fix indexing bottlenecks in days.