Vaex: Out of Core Dataframes for Python and Fast Visualization
▻https://towardsdatascience.com/vaex-out-of-core-dataframes-for-python-and-fast-visualization-12
Wouldn’t it be great if you could load a 1 TB data file instantly.
All this is possible with memory mapping, which is a technique where you tell the operating system that you want a piece of memory to be in sync with the content on disk. It is technically quite similar to a swap disk. If a piece of memory isn’t modified, or not used for a while, the kernel will discard it so that RAM can be reused.