Last year I had a garden for the first time. It was a raised garden made of wood pallets with slats removed, wrapped in garden clothe and filled with rich black soil from an old pig farm site. I looked under boards and rocks, found a buncha worms to add to them, too. I had three. A little after growing season, I added one little toad per raised container to eat bugs.
What if we added some mechanic to to add things to the garden that speed up growth? or added a chance to find a special ingredient if the soil had been tilled with "rich organic compost" bugs and toads added, etc.
My Animal pen is way ahead of my gardening and I keep needing crops I can't grow yet. I admit, I haven't made a chart to figure out the fastest way to lvl it ... but anyway...
The fastest way to level gardening is to grow exotics. If you really want to level as fast as possible (to the exclusion of all else largely) grow the highest level hasty exotics that you can, harvesting them every 15 minutes or 1 hour 40 mins as appropriate.