Published: May 09, 2025
Efficient data management techniques for green AI
Executive summary
This article delves into various techniques for efficient data management in the context of green AI, such as data compression, deduplication, and storage optimisation, while also examining innovations in data processing and management algorithms that aim to improve energy efficiency. The article outlines the benefits of adopting sustainable data management practices, the potential consequences of over-regulation in AI and data management, and the importance of task scheduling and resource allocation strategies to achieve energy efficiency optimisation in AI systems.
Introduction
As the digital and technological revolution continues to surge and expand, enormous amounts of data are being generated daily. It is estimated that by 2025, approximately 463 exabytes of data will be produced every day[1]. To put this into perspective, it would take around 237,832 years for a video conference to generate just one exabyte of data. This immense volume of data has led businesses to adopt eco-friendly approaches[2] and driven the creation of green data centres and sustainable data management practices. Green data centres are designed to employ energy-efficient technologies and practices to reduce energy consumption and carbon emissions, while utilising renewable energy sources. Sustainable data management refers to the processes and techniques used to collect, store, and process data in ways that minimise negative environmental impacts. By combining green data centres with sustainable content and data management, users can significantly reduce their environmental footprints for data storage and processing.
In 2024, we’ve witnessed an unprecedented whirlwind of interest in landmark AI technology. While the pursuit of 'big' AI persists, it is essential not to overlook the practical implementation of AI for everyday business applications. Accessing AI-driven insights requires time, resource investment, and substantial effort. It is vital to recognise that data, the driving force behind AI, is instrumental in propelling AI adoption. The cleaner the data, the more efficient AI solutions become. Data has the power to optimise these tools, ensuring that they reach their maximum potential.
However, the energy requirements of AI systems are an escalating concern, particularly with the growing utilisation of Deep Learning and other computation-intensive algorithms. This concern has become increasingly apparent with the widespread adoption of Large Language Models (LLMs) that rely on terabytes of data for training, that require considerable computing power and financial investment, and lead to significant carbon dioxide emissions. Consequently, a new sub-field has emerged to make the development and application of AI technologies environmentally sustainable: green AI[3].
With the expanding use of AI across diverse sectors, such as manufacturing, finance, and intelligent transportation systems, the leaders in the Asia-Pacific region (APAC) must urgently devise innovative data management solutions that can capitalise on AI's transformative potential while aligning with environmental objectives such as those outlined in the Singapore Green Plan 2030. Efficient data management is crucial for creating energy-conscious AI systems, as data-related operations like storage, retrieval, and pre-processing can consume substantial resources. The results of a recent study showed that a technique such as feature selection can reduce the energy consumption of AI model training by up to 76%, while preserving the performance of the model[4]. This article explores how data management can contribute to energy reduction through techniques like data compression, streamlined data pipelines, and other inventive approaches.
The environmental impact of conventional data management methods
Organisations process vast amounts of data to achieve their business outcomes, making proper data management and governance crucial to their success. As sustainability becomes a focal point in both business and technology, it is essential to design data management strategies with the environment in mind. Conventional data management methods often overlook the environmental impact of massive data centres, leading to an increased carbon footprint. These methods tend to be resource-intensive, consume large amounts of energy (often from non-renewable sources), and lack optimisation levers for usage, resulting in wastage of both compute and storage resources. Consequently, businesses face higher operational costs and struggle to reduce their overall environmental impact.
In data management, the impact is not only driven by compute provisioning factors, but also by practices adopted throughout the data lifecycle. Figure 1 shows three common challenges in conventional data management practices that lead to inefficiencies and wastage:
Figure 1. Challenges in conventional data management practices.
-
Isolated data and repetitive processes:
Data isolation occurs when data is trapped in separate departments or teams within an organisation, causing employees to re-do the same tasks, unaware of each other's progress. This results in data duplication and wastes storage, time, effort, and resources. -
Uniform approach to diverse use cases:
Using the same tool for all data management use cases leads to inefficiencies. Different use cases demand specialised solutions for optimal outcomes. For example, when you use a general-purpose database like MySQL for handling both transactional and analytical workloads, you may experience performance bottlenecks due to the diverse nature of these tasks. Instead, using specialised databases such as PostgreSQL for transactional workloads and Apache Cassandra for analytical workloads can lead to better performance, workload isolation, and an improved user experience. -
Inadequate data oversight:
The absence of data oversight directly contributes to resource wastage. Without proper supervision and standardisation, data becomes unorganised and duplicated, often leading to data quality issues and inaccurate metrics. Teams spend extra time locating the right datasets, moving data over the network, and correcting data inaccuracies, which requires more storage and processing power and could be avoided with robust data oversight practices.
Conventional data management practices can have a significant adverse environmental impact due to their resource-intensive nature and lack of optimisation. By adopting eco-friendly data management practices, organisations can minimise their environmental footprints, reduce operational costs, and contribute to a greener future.
Green AI and importance of sustainable data management
Green AI is a burgeoning subdomain that concentrates on creating and applying AI technologies in an eco-friendly manner. The primary goal of green AI is to address the energy demands and environmental consequences of AI systems, particularly those employing computation-heavy algorithms and vast quantities of data. A crucial aspect of green AI is sustainable data management, which is vital for minimising the environmental impact of AI systems and ensuring their responsible utilisation.
Sustainable data management involves the diligent control and treatment of data throughout its entire lifecycle, covering data collection, processing, storage, and disposal. The main aim of sustainable data management is to lessen the environmental effects of data management practices, reduce energy consumption, and maximise resource efficiency. Furthermore, sustainable data management focuses on guaranteeing that data is employed in a socially responsible and ethical manner.
The significance of sustainable data management is manifold. First, it assists in reducing the environmental consequences of data management practices. Data centres and other IT infrastructure are responsible for significant energy consumption and carbon emissions. Sustainable data management practices seek to lower energy consumption and carbon emissions by refining data centre design, boosting energy efficiency, and utilising renewable energy sources. Second, sustainable data management aids in decreasing waste and optimising resource usage. Data storage devices, such as hard drives and solid-state drives, consist of rare earth metals and other limited materials. Sustainable data management practices endeavour to minimise waste by ensuring effective data storage and processing, as well as responsible disposal.
Lastly, sustainable data management is essential for guaranteeing the socially responsible and ethical utilisation of data. Data privacy and security are critical concerns for organisations managing sensitive information. Sustainable data management practices aim to protect data from unauthorised access and usage while respecting individuals' privacy rights.
Sustainable data management is a fundamental component of green AI for addressing the environmental, resource, and ethical concerns associated with data management practices. By embracing sustainable data management techniques, organisations can contribute to the development of eco-friendly AI systems and encourage responsible data usage.
Techniques for efficient data management
With the increasing volume of data being generated every day, the need for efficient data management has become more critical than ever before. Techniques for efficient data management involve data compression, data deduplication, storage optimisation, and innovations in data processing and management algorithms (Figure 2).
Figure 2. Techniques for efficient data management.
Data deduplication is a technique used to eliminate redundant data by identifying and removing duplicate copies of data so that a single unique instance can be stored. Deduplication can be performed at the file level or the block level. File-level deduplication identifies and removes entire duplicate files, while block-level deduplication identifies and removes duplicate blocks of data within files. Data deduplication can significantly reduce storage requirements and improve storage efficiency.
Data compression is a technique used to reduce the size of data to make it more efficient to store and transmit. In recent years, many data compression techniques have been proposed for big data in cloud computing. These techniques include lossless compression, lossy compression, and hybrid compression. Lossless compression techniques, such as Huffman coding and Lempel-Ziv-Welch (LZW) coding, can achieve high compression ratios but have slower compression and decompression speeds. Lossy compression techniques, such as JPEG and MPEG, can achieve higher compression speeds but sacrifice some data fidelity. Hybrid compression techniques, such as Burrows-Wheeler transform and Run-Length Encoding (RLE), combine the advantages of both lossless and lossy compression techniques[5].
Storage optimisation techniques are used to improve the efficiency of data storage by organising and managing data in a way that minimises resource usage. Innovative data processing and management algorithms aim to improve energy efficiency by optimising data processing tasks, reducing resource usage, and minimising energy consumption. Examples of such innovations include energy-aware algorithms, adaptive data processing techniques, and machine learning-based optimisation methods.
Through advanced compression algorithms, AI systems can maintain data integrity while minimising storage and energy costs. Additionally, storage optimisation techniques, such as data deduplication and tiered storage, can further reduce energy consumption by minimising redundant data and prioritising energy-efficient storage solutions.
Best practices for sustainable data management
Sustainable data management is essential for creating green AI solutions that balance performance and environmental friendliness. By adhering to best practices in data management, organisations can ensure that that they minimise energy consumption and reduce the environmental impact of AI systems. The following best practices for sustainable data management in green AI should be considered (Figure 3):
Figure 3. Best practices for sustainable data management in green AI.
1. Energy-efficient data storage and management
- Enhance data centre design and energy efficiency while utilising renewable energy sources to decrease energy consumption and carbon emissions.
- Apply virtualisation, consolidation, and compression methods to diminish the physical footprint of data centres.
- Use data lifecycle management strategies to ensure data is stored efficiently to minimise the need for physical storage.
2. Data compression and storage optimisation
- Use advanced data compression algorithms to reduce the size of datasets without compromising data integrity.
- Implement data deduplication techniques to eliminate redundant data and minimise storage requirements.
3. Data pre-processing and feature selection
- Clean and pre-process data to remove inconsistencies, missing values, and outliers to ensure that AI systems process only relevant information.
- Apply feature selection techniques to pinpoint the most crucial features in datasets to lower computational complexity and energy consumption during training and inference.
4. Distributed data processing and Edge computing
- Utilise distributed data processing techniques to process data across multiple devices to reduce the energy consumption associated with data transmission and central processing.
- Implement Edge computing to process data closer to the source to reduce data transmission and latency and enhance energy efficiency.
5. Energy-aware data centre design and management
- Design and manage data centres with energy efficiency in mind using techniques such as free cooling, hot aisle/cold aisle containment, and efficient power distribution.
- Monitor and optimise data centre energy consumption through real-time energy management systems and predictive analytics.
6. Responsible data disposal and recycling
- Implement proper data disposal and storage device recycling to recover valuable materials such as rare earth metals and other limited resources.
- Use secure data deletion practices to protect data privacy and security during the disposal process.
7. Robust data security and privacy
- Establish strong security measures, including data encryption and access controls, to safeguard data from unauthorised access and use throughout its lifecycle.
- Emphasise data privacy by incorporating secure data management practices at all stages of data handling.
8. Environmentally conscious procurement practices
- Evaluate the environmental and social impact of procured products and services, considering the energy efficiency of IT equipment, the use of renewable energy sources, and the environmental impact of materials in data storage devices.
- Adopt eco-friendly procurement practices to minimise the environmental footprint of data management operations.
9. Data lifecycle management
- Implement data lifecycle management policies to ensure that data is stored, processed, and disposed of in an energy-efficient and environmentally friendly manner.
- Regularly review and update data retention policies to minimise the unnecessary storage and processing of outdated or irrelevant data.
10. Collaboration and open data sharing
- Encourage collaboration and open data sharing among researchers and organisations to reduce redundant data collection, and processing efforts to minimise energy consumption and environmental impact.
- Utilise data sharing platforms and data repositories that prioritise energy efficiency and sustainability.
11. Green data storage technologies
- Invest in energy-efficient storage technologies, such as solid-state drives (SSDs) and low-power hard disk drives (HDDs), to reduce energy consumption in data storage.
- Explore emerging green storage technologies, such as DNA storage and phase-change memory, which have the potential to significantly reduce energy consumption and environmental impact.
By implementing these best practices for sustainable data management, businesses can ensure that AI systems are optimised for energy efficiency while maintaining high performance and ultimately contribute to the development of green AI solutions.
References
[1] How much data is generated each day? https://www.weforum.org/agenda/2019/04/how-much-data-is-generated-each-day-cf4bddf29f/ via @wef
[2] Future of Green Business Through Sustainable Data Management, https://www.allerin.com/blog/future-of-green-business-through-sustainable-data-management
[3] R. Schwartz, J. Dodge, N. A. Smith, and O. Etzioni, “Green AI,” Communications of the ACM, vol.63, pp.54–63,112020.
[4] 2. R. Verdecchia, et al., "Data-Centric Green AI An Exploratory Empirical Study," in 2022 International Conference on ICT for Sustainability (ICT4S), Plovdiv, Bulgaria, 2022 pp. 35-45.
[5] T. Rajkumar Talekar, Efficient data compression techniques for big data in cloud computing: a comparative study, International Research Journal of Modernisation in Engineering Technology and Science, Volume:05/Issue:06/June-2023.