Your Smartwatch's Heart Rate Monitor Was Developed by a Furry

AI-Powered Intelligent Exploration Uncovers Hidden Meaning In Cybersecurity Data

As government leaders face a relentless artificial intelligence (AI) arms race against adversaries, a critical challenge emerges on the cybersecurity front: harnessing the power of data to secure government assets and thwart agile cyberattacks. AI is an essential asset in this data revolution, arming analysts with the tools they need to uncover threats before it's too late.

The world of cybersecurity is constantly evolving, creating hurdles for predictive AI models. AI can still play a pivotal role, however, as a partner for cyber analysts. Using AI-driven "intelligent exploration" to sift through high volumes of data, analysts can more efficiently detect anomalous activity, develop actionable insights and discover hidden connections in data.

IMPORTANT NOTICEAny information you supply is subject to our privacy policy. Access to this content is available to registered members at no cost. In order to provide you with this free service, the Government Executive Media Group may share member registration information with content sponsors.

How Generative AI Changes The Data Journey

As Deloitte has put it, data is "the new gold." Innovative IoT (internet of things) devices seem to arrive on the market daily, and the amount of data generated by these devices is exploding. Data holds massive power, and when utilized correctly, it can be extremely valuable for enterprises—both for improving business operations, and for improving IT operations. However, getting to that place where data is useful is a journey.

We see AI all around us and interact with it daily. As more and more enterprises figure out how to harness the data in their systems, the process is becoming increasingly easier and simpler. Data collection is the first part of the journey and is reasonably straightforward. But once we have collected all of the data, what do we do with it? How do we make sense of it all? How do you locate the specific information you are looking for in a data pile that rises as high as the sky?

Generative AI promises to make life dramatically easier on all of these fronts, across the enterprise. I'll focus here on what genAI can do for observability, devops, and IT teams.

Overwhelming amounts of cryptic data

Deloitte predicts that by 2025 our global data volume will reach 175 zettabytes, an increase of 55 zettabytes from where we currently stand. These overwhelming numbers can cause significant headaches for IT leaders as machine data can be cryptic and challenging to sift through.

Unfortunately, parsing this data is not as easy as reading a textbook, a magazine, or an article written by a human being. Often, when attempting to analyze machine-generated operations data, IT teams are faced with many unknowns—keywords, acronyms, numbers, codes—and need help knowing where to begin. I call these situations knowledge gaps. Like most people looking for answers, developers will turn to Google or other search engines to fill these knowledge gaps, which is time-consuming and unreliable.

Imagine how much better it would be if these knowledge gaps were quickly filled using generative AI. Generative AI has the potential to reduce toil for IT professionals by simplifying data and making it easily consumable.

How generative AI fills knowledge gaps

Another word for generative AI should be "simplification" because that is what it's all about. However, for generative AI to work its magic, it must be set up for success. Enterprises must strategically utilize generative AI within their systems; it cannot be overbearing or scary. I believe the best way to use generative AI is by keeping it as simple as possible and invisible to the end user. When implemented correctly, genAI should seamlessly blend into the workflow. The goal is for generative AI to reduce toil, not add additional stress, so making it easy to navigate is imperative.

When working with generative AI, context must be provided. Without context, AI is useless—similar to receiving ChatGPT information that only dates back to 2021. It's great to have access to mountains of data, but if AI does not have the proper context to sift through the data and find what you need, then the data will be useless and the AI will be irrelevant.

With the relevant context, generative AI can fill knowledge gaps in minutes, sift through hundreds of zettabytes in seconds, and provide fundamental information for IT and operations teams.

Generative AI in the real world

We see generative AI used in the observability space throughout many industries, especially regarding compliance. Let's look at healthcare, an industry where you must comply with HIPAA. You are dealing with sensitive information, generating tons of data from multiple servers, and you must annotate the data with compliance tags. An IT team might see a tag that says, "X is impacting 10.5.34 from GDPR…" The IT team may not even know what 10.5.34 means. This is a knowledge gap—something that can very quickly be fulfilled by having generative AI right there to quickly tell you, "X event happened, and the GDPR compliance that you're trying to meet by detecting this event is Y…" Now, the previously unknown data has turned into something that is human readable.

Another use case is transportation. Imagine you're running an application that's gathering information about flights coming into an airport. A machine-generated view of that will include flight codes and airport codes. Now let's say you want to understand what a flight code means or what an airport code means. Traditionally, you would use a search engine to inquire about specific flight or airport codes. Which city is the flight coming from? Where is the flight going next? These machine attributes are hard to read for a developer wanting to build a system that gathers all of this machine data using these machine tags. It is challenging to understand acronyms and numbers. Generative AI converts these acronyms and numbers into human-readable information that anybody can understand, making these systems more valuable for the average user.

These examples show the kinds of toil traditionally solved using search engines, knowledge boards, or repositories, taking hours to sort through large amounts of information. They are now solved with generative AI in a fraction of the time. This is a huge win for most enterprises, enabling self-service access to complex systems within the organization. This is empowering for organizations and their IT teams.

A more intelligent approach to data

Generative AI is still evolving at a rapid pace, and enterprises are still learning how to implement it into their data management systems. At Apica, we recently rolled out a generative AI assistant because, like most enterprises, our customers were looking to reduce the time and energy spent managing the massive amounts of incoming data.

While I currently believe that a generative AI assistant is the best way to use AI within data management, I'm not going to make any bets that this is the only way to do it. One thing I know for sure is that generative AI will not replace humans, but it will most definitely replace human toil.

Ranjan Parthasarathy is chief strategy officer for Apica, where he explores how generative AI can enhance observability, specifically using contextualized data to transform how devops and IT ops teams interact with their data. He was the founder of Logiq.Ai, recently acquired by Apica.

—

Generative AI Insights provides a venue for technology leaders—including vendors and other outside contributors—to explore and discuss the challenges and opportunities of generative artificial intelligence. The selection is wide-ranging, from technology deep dives to case studies to expert opinion, but also subjective, based on our judgment of which topics and treatments will best serve InfoWorld's technically sophisticated audience. InfoWorld does not accept marketing collateral for publication and reserves the right to edit all contributed content. Contact doug_dineley@foundryco.Com.

Copyright © 2023 IDG Communications, Inc.

Hack Geometry With AI, If Possible – New Data Methods

Video: What about shape data? How does that factor into new AI/ML models?

We've heard a lot – an amazing amount, actually – about data science applications for AI, with media like images, videos, music and all these other types of data assets – but what about geometry?

Justin Solomon starts out with a look at geometry and how it informs how we understand the world.

"As we navigate the environment around us, we solve geometry problems without even thinking about it," he says, giving some examples. "We figure out which side of a coffee cup will hold liquid, (or) the best way to grasp a door handle, and we navigate around furniture…"

Images

Justin Solomon

Helping computers understand shapes, Solomon says, will require some complex procedures.

"Elementary geometry provides a beautiful lens through which we can begin to understand what makes a shape unique," he says. "But there's only one problem. The shapes that we encounter in real life aren't just triangles, circles and platonic solids."

He talks about medical imaging, and about examples of "messy data" where data-driven approaches will likely prove effective.

"Consider an application of geometry to medical imaging," he says. "Given the shape of a person's brain from an MRI, can we identify which folds belong to which functional areas? This problem has one foot in geometry, since we're working with the shape of a brain, and another in data: experts know which fold is which thanks to centuries of experimentation and study. A professional can parcellate a brain because they've studied anatomy, and they've seen similar labels on other subjects."

(image caption: thinking about medical applications and the brain)

He also mentions lessons learned in handling data sets:

"The first question one might ask is whether we can just drop basic machine learning algorithms into our applications," Solomon says.

Part of the solution, he adds, is to reach a set of agreed representations for a data set.

As for common modalities, he discusses the use of mobile data, the ubiquity of text on the web, and other normal sources.

Common ML modalities

Justin Solomon

For example, he says there were over 1 trillion photos taken in 2022, and Wikipedia has over 4.3 billion words – these are examples of structured data that AI can work with.

Geometry, he suggests, is not as structured.

Citing a "paucity of clean geometric data," Solomon ponders how we can improve the study of this kind of spatial data.

"In early work, colleagues at the University of Massachusetts took virtual snapshots of 3d shapes, converting them to images that can be provided as input to 2d computer vision."

These sorts of tools, he says, can be "shockingly effective."

Take a look at the part of the video where he talks about three types of representations: point clouds, meshes, and CAD.

"Computer graphics applications prefer meshes, or networks of tiny triangles that are all linked together," he explains, contrasting the method to point clouds, which he called helpful in self-driving programs. "Mesh surfaces capture the smooth and detailed shapes of artists' designed characters. And then in computer aided design, we often prefer to work with big, smooth patches that are easily modeled by an engineer, and can be manufactured without facets."

Point clouds, Meshes & CAD

Justin Solomon

Specialized systems, he says, can be helpful.

"Think of a system that inputs a point cloud just as a long list of XYZ coordinates. Even in this setting, we have a number of demands. If we reorder our lists, the shape didn't change, so our model should be invariant to permutation. Moreover, different point clouds can have different numbers of points, which prevents us from using models whose input has a fixed size."

Solomon talks about the goal of mapping a shape to a meaningful input:

"One option is to adapt successful methods intended for other types of data, such as convolutional neural networks for images, but we can also devise specialized models for shape," he says.

One company that uses this sort of approach, he points out, is Waymo, where crowdsourcing involves geometric data.

Later, Solomon echoes a point that a number of our speakers have made: that self-supervised systems are the frontier. Addressing a key problem, that geometric data tends not to be annotated, he wonders aloud whether systems will be able to compile what they need.

"Self-supervised algorithms ask whether (the less robust data on geometry) is enough to train AI systems," he adds.

In closing, Solomon has more to say about this kind of specialized coding, and math:

"Although we haven't explored the details of modern shape analysis, hopefully you can see the challenges in teaching a computer how to understand geometry. At the very least, we have to deal with representation, modeling and data collection to make progress. Developments in this area will inspire intelligence systems that can navigate the road safely, bring digital characters to life, and manufacture shapes that fit our individual demands."

Portrait of Justin Solomon

Katherine Taylor

---