ELI5: n fold cross validation

What is n-fold cross validation?

Imagine you have a pile of flashcards you want to test yourself on. Instead of studying with some cards and testing with the same ones (which wouldn't show if you really learned), you'd split them into n equal piles, study with n-1 piles, and test with the leftover pile [1]. Then you repeat this process, using a different pile for testing each time, so every card gets tested exactly once [1].

Why do we do this?

When you train a machine learning model, you want to know how well it will work on brand new data it's never seen before [1]. If you test it on the same data you trained it with, it's like memorizing the answers instead of understanding the material—you'll get great scores but fail in the real world [1].

What actually happens?

You split your data into n equal chunks, then do n rounds of testing: in round 1 you use chunk 1 as the test set and chunks 2-n to train, in round 2 you use chunk 2 as the test set and the others to train, and so on [3]. At the end, you average all the test results to get a final score [1].

How do you pick n?

Common choices are 5-fold or 10-fold, with 10 being popular because it balances between having enough training data and being reliable [4][5]. If you have very little data, using a smaller number of folds means each test set is bigger and more reliable [5].