From time to time you do get large csv files. Take for example the open data from RDW with all the vehicles in The Netherlands. The size is ~ 7.6 GB (14 mln. rows and 64 columns), its not even really that large, but large enough for notepad, wordpad and Excel to hang….
There is a nice and handy tool XSV, see the github repo.
You can use it for some quick stats of the csv file and even some basic manipulations. For my csv file, It takes around 17 secs to count the number of records, around 18 secs to aggregate on a column.
In R data.table it took 2 minutes to import the set and in Python pandas 2.5 minutes
#CommandlineToolsAreStillCool #XSV #CSV
Longhow Lam's Blog 