The South African Radio Astronomy Observatory (SARAO) joined 30 other members in the establishment of the Ceph Foundation, to manage the massive growth in data and information generated from cloud, container, and Artificial Intelligence applications.
The Linux Foundation, a non-profit organization enabling mass innovation through open source, announced that over 30 global technology leaders are forming a new foundation to support the Ceph open source project community. The Ceph project develops a unified distributed storage system providing applications with object, block, and file system interfaces.
“Ceph has a long track record of success when it comes to helping organisations effectively manage high growth and expanding data storage demands,” said Jim Zemlin, Executive Director of the Linux Foundation. “This partnership will assist us to store and retrieve the huge volumes of data that will be collected by the MeerKAT radio telescope,” says Dr Rob Adam, Managing Director of SARAO.
The MeerKAT is a 64-antenna array radio telescope that has been built on the SKA site in the Karoo, and which will be integrated into the first phase of the Square Kilometre Array. MeerKAT has the capacity to process 275 Gigabytes per second in real time – that equates to approximately 58 DVDs per second. SARAO currently uses Ceph to synthesise a ~20 PB object-based storage system, for the data generated by the MeerKAT radio telescope array.
Ceph is used by cloud providers and enterprises around the world, including financial institutions (Bloomberg, Fidelity), cloud service providers (Rackspace, Linode), academic and government institutions (Massachusetts Open Cloud), telecommunications infrastructure providers (Deutsche Telekom), auto manufacturers (BMW), software solution providers (SAP, Salesforce), and many more.
“Ceph was designed and built for scalability, initially with supercomputers and later with cloud infrastructure in mind. A key design premise was that the storage system needs to provide a highly reliable and available service in a dynamic and increasingly heterogeneous hardware environment where everything can potentially fail,” said Carlos Maltzahn of University of California, Santa Cruz, a co-founder of the research project that first created Ceph over a decade ago.
Ceph is also used by Rook, a Cloud Native Computing Foundation project that brings seamless provisioning of file, block and object storage services into the Kubernetes environment, running the Ceph storage infrastructure in containers alongside applications that are consuming that storage.
Efficient, agile, and massively scalable, Ceph significantly lowers the cost of storing enterprise information in the private cloud and provides high availability to any object, file, and block data. Block and file storage are critical to any IT infrastructure organisation and are important components of infrastructure platforms like OpenStack and Kubernetes.
“SARAO uses Ceph in concert with locally manufactured hardware to lower storage capital expenditure for the MeerKAT storage infrastructure. In order to share our experiences and showcase Ceph, the SARAO storage team is in the process of establishing a Cape Town Ceph community forum,” says Thomas Bennett, Senior Software Systems Engineer at SARAO.
The growth of new cloud, container and artificial intelligence/machine learning applications are driving the increased use of Ceph. Ceph combined with analytics and machine learning can, for example, enable enterprises to comb through mass amounts of unstructured data to quickly spot patterns in customer behaviour, online customer conversations, and potential noncompliance scenarios.
“A guiding vision for Ceph is to be the state of the art for reliable, scale-out storage, and to do so with 100% open source,” said Sage Weil, Ceph co-creator, project leader, and chief architect at Red Hat for Ceph. “While early public cloud providers popularised self-service storage infrastructure, Ceph brings the same set of capabilities to service providers, enterprises, and individuals alike, with the power of a robust development and user community to drive future innovation in the storage space. Today’s launch of the Ceph Foundation is a testament to the strength of a diverse open source community coming together to address the explosive growth in data storage and services.”
The inaugural meeting for this community forum will take place on the 28th of November at the V&A Waterfront in Cape Town.