Dec 19, 2017

Read and write using fst & feather for large data files.




For past few years , I was using feather as my favorite data writing and reading option in R (one reason was its cross platform compatible across Julia, Python and R), however, recently, observed it’s read and write time lines were not at all effective with large files of size > 5 GB. And found fst format to be good for both read and write of large files in R, only disadvantage is it not cross platform compatible as feather. Especially, fst compression with 100% is good for storage of large files and it is more efficient in reading the same into R environment. Also, I feel no point in re-testing and giving benchmarks as they are already available at fst site.


                                                                     ------ Happy R programming let me know your experiences.