FSISUS11: Do you store processed data or raw data?

FSISUS11-BP01 Use processed data to reduce your storage footprint

Often raw data from your data sources may include a large number of observations from streaming data sources that continually produce data or include large amounts of redundant data from a variety of sources. You can reduce your storage requirements by first processing the raw data, then storing only the results. Unless you have a raw data retention compliance policy or requirement, you can purge the raw data automatically shortly after processing to reduce your data storage requirements.

Store processed generative AI training data rather than raw data when compliance allows. Implement efficient vector storage strategies for generative AI applications. Optimize vector lengths for embedded tokens in generative AI systems.