单细胞数据爆发，现在细胞数据随便就是万和百万级别，除了有许多专门开发出来快速处理百万级的工具包（例如Cumulus等）和down sample的思想外，为了减轻个人PC内存有限的问题，其实方法还蛮多的，例如ArchR处理scATAC数据一样，就是将数据放在硬盘里面调用，而不是全部一股脑读入内存，这种思想在大数据处理时还蛮常见的。
以SCE（SingleCellExperiment）对象为例，主要实现方法就是HDF5Array包，将数据以h5的格式存储在硬盘上，然后sce对象灵活访问和调用，减轻内存负荷。
In the context of a SingleCellExperiment object, the file path linked with the assays can be used to efficiently manage large datasets by storing the assay data on disk rather than in memory. This is particularly useful when dealing with large single-cell RNA sequencing datasets.
The HDF5Array package in Bioconductor allows you to store assay data in HDF5 files, which can then be linked to a SingleCellExperiment object. This approach helps manage memory usage efficiently. Here's how you can create a SingleCellExperiment object with assay data stored in an HDF5 file:
1. 工具包配置:
if (!requireNamespace("BiocManager", quietly = TRUE))
install.packages("BiocManager")
BiocManager::install(c("SingleCellExperiment", "HDF5Array"))
library(SingleCellExperiment)
library(HDF5Array)
SingleCellExperiment对象本质也是R的S4类对象，和Suerat object还有scanpy的anndata都是可以灵活转化的，非常简洁。
SCE对象的官方教程：Chapter 4 The SingleCellExperiment class | Introduction to Single-Cell Analysis with Bioconductor
2. 创建基因x细胞的表达矩阵，且存为h5文件
# Create some example data
# 如果你有 这里换成读入
counts <- matrix(rpois(100, lambda = 10), nrow = 10, ncol = 10)
rownames(counts) <- paste0("Gene", 1:10)
colnames(counts) <- paste0("Cell", 1:10)
# Save the counts matrix to an HDF5 file
h5file <- tempfile(fileext = ".h5")
writeHDF5Array(counts, filepath = h5file, name = "counts")
如果是10X之类的数据，对应读入就好了，不需要自己手动构建。
3. 创建SCE对象
# Load the HDF5 file as an HDF5Array
counts_h5 <- HDF5Array(filepath = h5file, name = "counts")
# Create the SingleCellExperiment object
sce <- SingleCellExperiment(assays = list(counts = counts_h5))
# Add row data (gene metadata)
rowData(sce) <- DataFrame(GeneID = rownames(counts))
# Add column data (cell metadata)
colData(sce) <- DataFrame(CellID = colnames(counts))
# Accessing the data
counts(sce)
rowData(sce)
colData(sce)
In this example, the counts matrix is stored in an HDF5 file, and the SingleCellExperiment object links to this file for the assay data. This approach is particularly useful for large datasets, as it avoids loading the entire dataset into memory.
You can access and manipulate the assay data in the SingleCellExperiment object just like any other assay, but the data will be read from and written to the HDF5 file on disk as needed.
然后你就会发现，你可以随意访问counts矩阵，但是这个sce对象却非常小，具体来看，其实会发现它内部的sce@assays@data@listData[["counts"]]@seed@seed@filepath其实只是一个矩阵路径，而不是真实的数据。
