00:00
Thank you for joining us for our API session. So I'm Nihal Mirai, uh Solutions Marketing Manager for Databases. I'm joined here with Anthony. Um Wanna introduce yourself. Oh, sure. Hey, uh Anthony Nocentini, I'm a senior principal field solution architect,
00:15
uh focusing on databases, cloud automation. And so we're gonna talk a bunch about automation today. Awesome. All right. So really simple agenda. We'll just spend five minutes talking about the API overview P OS approach to API S. Anthony is gonna spend um most of the session on the tools and integration aspects and
00:37
then definitely spend a lot of time on the demos. So let's get into it. So just like every organization or every, every vendor out there when we're designing API S, we obviously want our customers to use it. Um So we talk to a lot of personas and they all have different needs,
00:57
right? Whether it's the storage admin, whether it's a database admin coin architect, they all have different needs. And so uh it's about trying to get to design uh one API that can be consumed, consumed by all. And um the, when it comes to API first strategy, it means different things to different
01:16
stakeholders. Uh When it's, when the organization is actually looking at uh an API first strategy, it's about formal life cycle designs. It's about essentially having the right documentation when the API is changed or if it's sunset. Uh It's about making sure that the user community is informed.
01:37
So it's uh they're coming from it from a different angle. Then when it's a cio obviously, they're concerned about integration and making sure that it doesn't break other pieces of the, the environment. When it's a developer, they're looking at it. Sometimes they think about API first as developing the A P first and then developing
01:59
the features when it's really about user experience. So that can cause a little bit of a mismatch. And then the last is when we talk to site reliability engineers, the feedback we get is their role is typically orthogonal to the A PS themselves. But they are concerned about metrics.
02:19
How do you measure when things go wrong? How do you kind of improve or how do you troubleshoot? Right. So all of this feeds into the A P first strategy. Now, some of the common challenges with the A P first strategies that we've seen. One is certainly from a design perspective. A common pitfall is when vendors design the feature first and then they bolt on the
02:44
API that causes an impetus mismatch because obviously they're called cloud operating model and the A PS for that work. Differently. Um Communication is a, is a big concern when you have two teams that are actually developing the API S and they're not communicating, you can run into all kinds of issues. Documentation is another big thing when the documentation does not match what the API is
03:12
doing, that can cause a lot of dissatisfaction. Uh We touched a little bit about telemetry and the fact that, you know, you can't improve if you can't measure. So that's really a key aspect of uh A P strategy. And then of course, it has to support different tools and languages in our session.
03:29
Today, we're gonna show you demos around powershell, but obviously, there's a lot of other languages and tools that have to be supported. And the last piece is around authentication mechanisms, right? Um uh On Prem and in the cloud have, have different uh mechanisms that have to be supported. So pure, recognizes all of this.
03:51
Um And we have an api first strategy. Um And we have customers using our API S in different ways, whether it's for automating, uh you know, their data management, you know, snapshots, taking snapshots um from a monitoring perspective, using it for monitoring real time and being able to resolve proactively some of the the
04:13
issues. Um We, we see them using it with orches orchestration platforms, whether it's answerable or any of the other uh platforms out there. Um And certainly our, our vision is to kind of provide that common data experience, whether it's on prem or in the cloud. Um from a data protection perspective, it's, it's about actually being able to automate some
04:38
of the backups and restores. And last, but not the least, it's, it's about kind of from a support angle for virtualized environments as well as co environments. So that is really where we are driving with our A PS Anthony. I know you, you work with a lot of customers and so why don't you share some of your insights?
05:01
So going through kind of the big pillars on the slide here, this is where we spend most of our time in the field, talking to customers about the integrations that they wanna build. I spend the majority of my time uh in the middle top one on monitoring and also the bottom right one on cloud integration. And we have customers that are starting their cloud journey and they're looking at how do I
05:21
take something that I built on prem and leverage that in the cloud? And so that's kind of one of the core things that ni hall touched on is kind of having that hybrid cloud experience because it's on our platform, whether it's flash array or cloud block store, it's the same API so we have customers that will build workflows and integrations on prem.
05:40
And they think, well, how am I gonna use this in the cloud? Well, OK, let's just go ahead and use that in the cloud and cloud block store. And so being able to move that transition goes back to helping our customers not or ma manage that risk over time, right? If I'm ac level person or direct level person, it's gotta make a platform decision and how that's gonna be supportable.
05:58
That's an extraordinarily valuable uh piece of information. So we're gonna start from zero today about what is the rest api talk a little bit about our powershell module and then we're gonna spend probably 37 36 minutes in demos. Um And I wanna make sure that you have the code to go after this and go play with this when you go back to wherever you are. So who's heard of arrest API?
06:20
Right. Who's the developer? OK, cool. So that's, so I wanna do a quick tour of what rest API is and then we'll go straight into it because I'm gonna call out a couple of those things. Once we get into the demos, the idea is we can interact with the resource over HTP. And it's become the standard for systems that
06:37
wanna exchange state in a stateless way. I wanna be able to talk to a system and be able to do the things like be able to create update or delete resources. What's interesting is it is stateless. So you'll find uh that you have to query a resource, see if it's there and then operate on the resource and we'll do that a bunch today. How we're gonna interoperate with those things is just over HTP.
06:58
Right? And we're gonna do that. There's a text representation of the object that we wanna interact with in the platform. So today we're gonna look at things like hosts and volumes and performance data and a bunch of things that we can pull out of our arrays. So yeah, touched on the kind of the crud operations of it and using kind of the predefined HT methods
07:17
of get post patch update, things like that or the way that we're gonna interact with these things. So taking a system like HTP and leveraging that to be able to kind of manage state in us in a system has been an extraordinarily powerful concept to go to do those kind of operations. One of the things I do want to point out is that API S are versioned and what that means is
07:41
independent of the version of purity or even with any operating environment, the API is gonna be versioned independently of the actual operating environment purity in our case, uh or other types of API S that you might be using. And the reason for that is if you build an integration and the object looks one way, well, if we come along and change the functionality of the object in a newer version,
08:04
well, we don't want to break your previous version. So we'll stabilize the object or we'll stabilize the object's definition within an API version. So you have code stability against that thing, right? So we won't break your code going forward if you, we move the API forward in some way, or anybody moves their API forward in some way.
08:21
And so we'll get into how to navigate that in our platform. So like ne Hall said, uh we have lots of integrations across lots of different products and platforms and communities and things like that. And it really the foundation of that is the fact that pure is taking an api first way of inter operating with other things, right? So I think VM Ware Cisco inter site docker
08:43
containers, the ability to integrate with all of these things over rests and kind of meeting the the community and the other um ecosystem that is uh the integrations that we wanted to operate with over rest, right? The industry is kind of stabilized on that. So today, uh as ni Hall indicated, everything is backed by rest, right? And how we're gonna interoperate those things.
09:07
I'm a powershell nerd. I don't know a lot of Python Copilot and I have been spending a lot of time together so I can learn how to write Python. But the idea is, right? The idea is this is a tool that talks rest, right. Python is a tool that talks rests and we expose that functionality to you to meet you where you are in your automation journey.
09:27
So the stuff that we're gonna do today, I'm gonna do in powershell. You certainly could do it in Python. You could certainly do it in direct rest. Uh One of our customers is here and they're doing that exact thing right now in their data center is inter operating with our API over direct rest. And the reason why I like to use powershell is I have a pretty strong Microsoft background,
09:44
but also it's one of the most widely used integrations that we have at pure, right. And so our Powershell module has command, lets it go across all of the different objects that are available inside of the array and currently supports uh rest version 1.02 0.33. That's our rest API versions inside of purity.
10:05
Uh We have two different versions of the powershell module out there, the original SDK and what's called SDK two. And we're trying to get our customers to move the SDK two so they could take advantage of the version to API and the functionality that it has. Uh we have cross platform support, which is really cool because I have a Mac and I can do
10:24
the things that I wanna do in the Microsoft ecosystem. And we also have a bunch of demos uh that are available on our blogs, our website and on github, which we'll get into around how to do pretty sophisticated things, leveraging our platform inside of uh Power Cell cool and So the uh last time I checked about 40,000 downloads for the Power Shop uh module,
10:49
which is pretty, pretty good in our universe uh in our customer base. And so we see that that gets uh effectively used. So now four minutes longer than I thought before I got into the demos, but here we are. So I'll make the code available to everyone. Uh And I'll put a link at the end and it'll come out with the conference.
11:07
And so what we're gonna do is we're gonna go through a bunch of demos today, just kinda navigate what the power Shaw module can bring to you. And again, remember this is just rest. So all of these things can immediately transport over to Python. The Powers Shaw module itself is the pure storage powershell SDK two.
11:24
And the first thing that we want to do is I had a really hard time typing my password yesterday. So I decided to not type my password interactively and just load it from a credential file. Uh So there we go. So when you connect to a flash array with the, connect to uh P fa two array command lit, you find out where your array is and I'm just gonna hit it by a DNS name pass in a credential.
11:45
I have a self signed certificate. Don't tell our security people and I'm gonna use this verbose parameter uh which is gonna help me discover some pretty cool things behind the scenes. And so when I run this code, what it's gonna do since I didn't specify an API version, that one right there is commented out it's gonna negotiate to the highest API version that is available in purity on that particular flash
12:08
array, which is 2.33. If I needed to, I could specify an exact API version if I wanted to, to maybe go back to a previous definition of an object, right? So that might be something that you'd be interested in. Uh When you're building an integration, I do want to call out.
12:25
Then I add also when I add the verbose parameter. If I go way up to the top, you'll actually see the rest API call against the endpoint, right? I'm using the HTP verb, get against that API end point. And then you can see what's returned to me is the JSON representation of that end point, right?
12:43
I happened to just be uh an array of versions that are available to me and you saw it kind of ripped through all those when it printed out all the different versions here in the output. So with that, let's go ahead and take our connection and store that in this flash right object here because I'm gonna use that over and over again to communicate to our API and kind of revisit that concept that this really is rest behind the scenes.
13:07
One of the more powerful commands that we have is the ability to just talk direct rest to the API in our power Shaw module that handles all the authentication. So I don't have to pass in bearer tokens or any more complex constructs. And so this command here invoke P fa two rest command allows me to define a method get against the UR I, which is the thing that I want.
13:29
And so this just so happens to be the same exact call that you saw a second ago when it was output all the API version. So I just literally just did the same thing of being very verbose about how I want to interact with this. And we can also do this against many other end points just to highlight that that capability is there. And the reason why I'd want to do this is
13:47
perhaps I am building a more complex integration outside of powershell. And I wanna kind of discover more simply what's available to me. I don't like reading documentation. I like going and like breaking things and then figuring out what it actually means, spending six hours to do that to finally go back and read the docs. But you know how that works,
14:02
right? So exactly that in you can see as you get more complex representations, I'm just gonna go ahead and talk to you an actual volume. And so I can specify the volume that I wanna talk to you in a format that looks like this and that's just a VV name where I'm gonna hit the end point and go after that exact object name. It says give me this exact volume and pull that
14:23
back in Jason. But this seems like it's kind of hard, right? I don't want to be doing this. Uh as I'm starting to build uh in integration, I wanna go maybe stick with the powershell and leverage the commands that are available to me. We have hundreds of command letts against all the different API objects that are available and we kind of classify them into big buckets.
14:41
We can talk to things about volumes and arrays and performance. And so if I use powershell to give me all the commands that are available, you'll see we have a pretty good chunk of stuff to interoperate with volumes. Perhaps I want to remove a volume or create a volume snapshot or create a new protection group. These are all different operations that are
14:58
exposed to me to interact with our array. Well, where you are, right, hopefully, right. All right. So get P FA two volume, which we were operating against the endpoint itself and we saw all the data that's coming back as JSON. Well, we have a command that wraps that up. And one of the things uh in our system is I can
15:22
do a get P FA volume and it'll give me all of the information about those volumes of it. It'll give it back to me an object format. So I'm gonna deal with this in a native powershell way now um for most of the rest of the demos, but I do want to point out that this particular array has 743 volumes, right? And so if I go and do kind of a traditional
15:42
power cell experience of, give me all the volumes go across the pipeline, expand the property space sort by descending for the total physical size, give me the top 10 and form at a table. So this is probably a pretty common thing that uh an administrator would do in an array. I wanna get all of the top 10 volumes by physical space and this is the standard power show way of being able to do that on our system.
16:06
And so what happens right now is it's gonna grab all 800 of those objects like 743. Was it? Yeah, and pull them across to my local system and then do the work on my local pipeline on the client. Ok. Well, in our flash array, the API has some pretty cool functionality to help you do operations like that in a very efficient way.
16:27
And so what we have is the ability to push those operations into the array. So what I'll do is we'll do a sort, we'll pull down a 10 and just return 10 rows to my local clients at a 743. And so with all of our command letts where you're retrieving information, they have the ability to do things like sort what I have here is I'm gonna sort by space dot
16:50
Total physical and then a minus sign. So let's unpack that a little bit. I read the API Docs and that, that link there. You'll find that. It's kidding. Um, I did read the docs once. Uh So this link here, you'll find the actual object definitions and you'll see that underneath the space property there is a total physical property.
17:07
And then if you put a minus sign in there, it's gonna sort it by descending, right? And then just limit top 10. And so I do this now and I get the same exact output. I get the name of the volume, it's data reduction, the shared space snapshot, then provisioning and total physical, sorry that it's running off the side of the screen
17:26
there. But I had to fight for screen resolution from a readability standpoint. So let's do something. Let's measure how long that takes. Let's take the first command, let and run that against and measure command to see how long that takes. 566 milliseconds.
17:41
Let's do the second one where we got the same exact data, but it pushed all the work in the array. It's about five times faster, right? Because all of that's happening on the server side and I'm just returning 10 rows. And so when you are building the API integration, that's kind of how you want to think about how to interact with the array is
17:59
let the array do the hard work and return back to the client which you really need. And so we're gonna exploit this fact and do some pretty cool things over the rest of the demos here. That was a fantastic pun, by the way, uh In addition to uh sorting, we also have filtering which has become a very common operation. Traditionally in powershell, you'd handle that
18:19
in this way where I'd bring all the things locally, throw it across the pipeline. And I wanna find volumes that have this string representation in the name. Well, we have a filter option here and our filter clause is structured where we have the name of the object. And then I have to do some fun uh quoting to make sure that that's treated as ill literal as it's passed in.
18:41
And I'm using doing a wild card search to give me all the volumes that have that string, right? And then that happened all on the array side and return those objects to me on this local system. And so the kind of the key takeaway there is let the API do the heavy lifting, right? Don't build the heavy lifting into your clients
18:58
that application because this is gonna start to get troublesome at scale, right? When you start talking about I only have 743 volumes. What do you have 7000 or what if you had 30 arrays that have 7000 objects? I'm gonna let the arrays do the heavy lifting so that your integration can be kind of lean
19:13
and mean. So any questions or comments about filtering API S we gonna keep going. Yeah, exciting stuff for like three and 15 in the afternoon. Look at all this code. Uh But the key here is I want you to take this, pick it up,
19:28
go back to your workplace or home and, and do these things in your lab. So let's kick off a workload. Uh What I'm gonna do here is I'm gonna start a job that's gonna drive a database back up on one of my systems in my lab. Uh So we'll get some performance data to look at. Uh I spend a lot of time in the performance universe.
19:48
I'm a database person by trade. I need to make database servers fast, which is generally pretty easy on flash arrays. Uh But sometimes I do have to do some performance troubleshooting. And one of the things I wanted to empower me to do our customers to do. And our other folks in the field and support is to slice and dice through performance data
20:05
really quickly, right? And when you work with a object, especially uh the volume host and I believe volume group, they all have uh performance metrics associated with them. And some of these metrics are probably familiar to you if you've worked in flash array, things like service time or SAN time are in um mirrored rights.
20:28
So the kind of all those things that you've seen in a flash array gooey are available here inside this object that's returned that I'm going to interoperate with. And so we also have things like um right bytes per operation or the size of the right or uh most of rights per second, things like that. And so we're able to interrogate very granularly the performance data on an object. In this case,
20:51
it's a volume also could be a host, which I think I have some code for that. And so this is where it gets to be really important to do some of the API side filtering. So if I wanted to ask the array, give me all the top 10 volumes sorted by reads descending, right? Basically, who's the busiest thing on this array? And I wanna know and it's gonna go and run that
21:12
code and return that back to me pretty quickly, right? That's a couple milliseconds to run that to run that query. And so here we can see that I'm getting a pretty mild workload on that particular volume, but very quickly, you can identify a uh busy volume inside the array. But we saw in the object model we had reads and writes.
21:33
Uh So there are times where we have to do some heavy lifting to get a different representation of the state of the system because if I just said, give me all iops. Well, we're not surfacing that to you as a metric because we're surfacing, reads and rights. So in this case, we have to do some math on the client side, unfortunately, on a client side. And so what I'll do to help me with this one is
21:54
I'll grab the performance data and then I'll do the math on the client side to do what I wanna do to find things like total iops. And so just some kind of some syntactical power. So I'll go over here just adding those two attributes together, labeling that as I OS per second. And again, I can go and after and find very quickly the busiest hosts on an individual system.
22:17
But when I ran that code, it looked at a 32nd window from now back, right? But what if I wanted to answer a question, what happened three days ago or two days ago? And I wanna do that kind of forensic troubleshooting at the command line on our system. Well, we certainly could go into Google and you could expand it back over two days or things
22:34
like that. But what if I needed to do this across multiple arrays? Well, that's where you'd leverage integration like this. So jumping ahead, I wanna go and do some time zone math which has gotten me in a lot of trouble in my career because I'm really bad at time zones. And so I do this all on code here.
22:50
I'm gonna go back and say, you know what, give me a range uh for 24 hours starting two days ago, that what's happening there. And so when I ask for the volume performance, same code that we saw a second ago, I'm gonna sort by Reed's descending. Give me the last 10 or top 10 and I can pre pump in the range there and also define a resolution which is a sample interval across
23:12
that range. And then I can answer the question, who were the highest read volumes in that range two days ago? OK. So we go back in time here right back to the future uh on 617 because today I believe is what it is today is the 20th, right? Yeah.
23:30
So never type in demos. That's a presentation tip, right? Bad things can happen. But also I can start to get build really complex structures here. I can do things like get um get the performance limit to the top 10 over a range and then combine all the filtering that we talked about to limit to specific workload.
23:51
And so right here, I'm going after any volumes that have that string and I can go and collect up potentially all of the sequel servers. But this requires me to have a naming convention that would involve sequel in the name for me to, able to identify those. And we're gonna talk about a technique using tags in a minute where we can do something like that a little more accurately than string
24:11
matching. And as I indicated earlier, it's not just for volumes also for hosts. So I can go and see who in our lab is a, a big reefer. There's not a lot of workload going on today in our lab, probably because we're all here, right. Yeah.
24:30
Uh But the same concept there a second ago, we looked at volumes here, we're looking at host performance across all hosts attached to an array. And so the main idea here is leveraging uh our API to go and slice and dice through this performance data in a uh very uh that's what I'm looking for uh efficient way. One thing I do want to call out and I do this a lot uh is if you are building an integration on
24:56
rest and we have a pretty complex powershell command up there. If I add the verbose parameter, it's gonna give me there's the JSON that's returned, you saw the structured output, I scrolled past that pretty quickly. But if I also, if I look closely here, I have the actual API end point with all the parameters populated. And so that means again,
25:21
I'm not the best API rest program on the planet, but I'm pretty good at powershell. So I can use this to help me discover some pretty complex representations. If I'm building a more sophisticated integration outside of powershell or in Python? Cool. Awesome. Any questions or comments, team? Exciting. I mean, yeah, I mean I told you I was a
25:42
performance junkie, right? So here we are. All right. So tags, um who's leveraging tags in their infrastructure today, right? Mostly in the cloud, right? But we took that and brought that into flash ray. Did you know flash ray had tags built in?
25:56
Are you using tags? Anybody? Ok. So here's the way that I use tags uh in our environment. So what I'm gonna do is I'm gonna build two sets of volumes and I'll walk through this code here in a second. I'm gonna get a listing of all the volumes that have that string match. So basically, I have two big buckets.
26:15
I have all the volumes attached to SQL server A, I'm gonna have all the volumes that are attached to sequel server B and I'm storing them in those this variable right here. And if I go and I output that just to make sure that I have data, we can see all those volumes, right? Cool. I'm gonna build up uh a tagging protocol, I guess you could say uh to help me
26:37
assign some additional metadata to those volumes so I can do some more complex things. We have a concept of a name space which is kind of like a folder. I, I hate that metaphor when we talk about name spaces, but it's the ability for me to just say, you know what these are, all the things that belong to Anthony over here in this corner and what a tag is, is a key and a value pair. And so I'm gonna have the key B SQL instance
26:59
and I'm gonna assign a tag value of the actual server name, which is A E and SQL 22 A and B, right? I just set up some variables here. I didn't actually do anything until now. So what I'm gonna do is I'm gonna use this command lit here. Uh set P fa two volume tag batch against the flash array.
27:18
Uh There's our name space variable that we just built up. The resource names are the two buckets of volumes that we just built and then add in the tag key and the tag value. And so what we're just doing now is for all of those volumes, we're assigning kind of our key value uh tagging protocol to all of those. And so we'll see things like uh key. There's the name space,
27:42
the resource id is a unique identifier for the actual resource. That's not its string name, there's the string name and then there's the value. So if we scroll back, we'll see many of those were populated during that operation and it needs to go back even further. We can see the ones for a EN SQL 22 A, right? Cool.
28:02
Well, let's let the API do what the API does. So now I can start interacting with those objects based off the tagging scheme, right? Not some string matching, which might be not maintained over time when it comes to a uh provisioning standpoint. So we usually we'll see customers do is when a provision volumes,
28:22
they'll inject this into that part of the infrastructure to the tagging mechanism is associated with that. And so now I can start operating on these things in a way as collections of those objects. So get P fa volume tag will go and ask the array to give me all of the ones that have that particular key sequence. And so very quickly, I can get a list of all the volumes that are associated with SQL
28:47
servers. OK? Or any other workload doesn't have to be SQL but it's kind of my jam, right? Cool. So when I returned that list, I have this object that's returned, it's got those parameters or excuse me, properties, copy key name,
29:03
space, resource and value. If I crack open dot resource dot id out of there, I get all these goods that represent those volumes. OK? Now I can do things like hey get P fa two volume space for an array. I can operate on that on the subset of things.
29:25
So very quickly I can get exact data about those exact things for SQL servers or whatever workload it is that I wanna tag. And so in this example, here, I'm going and asking the array, give me all of the volumes uh sort it now negative or sort of descending on data reduction. So now I have two volumes that of 1.0 data
29:44
reduction. Is that good or bad, bad, right? And so I could potentially have a report like this that pops out, I could have an integration that triggered something maybe a DB A turned on TDE and this is where we are, right? This could put us in a bad position from a provisioning standpoint.
29:59
But the core concept I wanna highlight here is being able to very quickly get to an exact subset of performance data about a thing. It doesn't have to be data reduction. It could be any of the other performance attributes that we've looked at. OK? Cool. So doing that again across uh volume performance.
30:17
I can say, you know what I just wanna focus on the sequel server volumes. I'm not gonna do all that string matching stuff. I'm gonna pass in that array of goods that we have and it'll go and do that exact thing again. But for the performance data kind of leveraging the API to help us get and slice and dice that information. And when we're done, you don't have to delete your tags,
30:42
but I have to delete these tags because I wanna come back and do this demo again sometime in the future, right? So any questions or comments around tags that seem like a useful integration to build some intelligence into your applicant into your infrastructure. So you can answer some hard questions, right? Always blame the DB A S.
31:03
No. Uh The question is, are tags just for volumes. No, you can do um against hosts and other objects as well. I don't remember the whole list at the top of my head, but it's not just for volumes. Yeah. Oh, good question. So who has an array with a lot of snapshots?
31:21
Who has an array with a lot of snapshots that you wanna clean up, right? Without having to click through the gooey. OK. So the same concepts that we've been driving, but just a different context here is I'm gonna work with a snapshot and I'm gonna do some things with that. And so when I take a snapshot,
31:37
uh we have the snapshot name of a protection group, there's an auto incrementing integer, there's also a resource identifier and some other information that we can use to work with that particular object. And if I look inside of here, I do get member, we can see again all the different properties are available for me to go find things about that particular protection group snapshot.
32:01
One thing in the data space that I work a lot with customers is kind of embedding. Some, let's just say application specific knowledge into a snapshot, right? We saw the auto increment, the integer. Does that mean anything to a business process? Right. The fact there was like 1667. It was very awkward yesterday because it was
32:18
like 666. I very quickly got through that demo with out any issues. Um But years ago, somebody uh on our development team put the suffix parameter uh on a snapshot and so I could do something like inject some application specific knowledge into a snapshot. And so in the data space, if I do something that takes a long time and it breaks,
32:41
do I want to revert the snapshot all the way back to the beginning? Or can I go somewhere in the intermediate phase and checkpoint the operation and just roll back to that point? Right? And so we see lots of folks that do data warehousing uh ML training and modeling, use this functionality to checkpoint a business process as it goes through.
32:59
So you can revert back to a known good state, not having to go all the way back to zero. So that's one way to do this is by adding a suffix. And so now when I come back later, I can go and I can ask the array, give me that exact snapshot because I need to revert back to that known state, right? And so that can be built into the data side
33:19
when you get to that point, make the API call or powers shall command the call or Python call to do that thing. And you'll be able to have that as part of a business process or data load. And I can very quickly just ask the array again using the filtering techniques uh that we've talked about filter against the suffix and I can get that exact snapshot returned,
33:41
right? Because other techniques that I've seen folks use is the sort at the most recent one, but that might not be bound exactly to the process that you want to have in the system. Cool. Another common thing that we get asked to do is clean up old snapshots. So again, we're gonna do some date math here, go back 30 days.
34:01
I'm gonna ask my array, give me all the snapshots that are older than 30 days, sort it by create a date and so I can go pretty quickly through and get a big bucket. Clearly, we're not cleaning things up in our lab. Good morning, by the way, good morning. Uh So we have most of them are me, by the way, if you didn't notice it is a shared lab.
34:23
So lots of volumes there that I need to probably do some clean up on. But I don't and you can also do this for protection groups. This was volume snapshots. This is a protection group. So same exact concept to get that bucket of things. Now, uh I did find this a little bit.
34:39
Uh That's what I'm looking for. Not cool. Maybe friction full is the, maybe a, a verb for that. I would expect that I could do something like remove uh P fa two snapshot and then pipe that into or get my snapshot list and pipe it into the command lit to delete that we can't do that. We can't pass that object across the pipeline.
35:00
So we have to use the API technique of getting all the protection group snapshots that we want that match uh that um match for our protection group store the object. And then we're gonna work on the resource ID. So rather than do you know, get the protection group snapshots and pass it across the pipeline, we'll use the API technique of just passing in the ID.
35:26
And I leave this commented out because I don't like to leave potentially destructive code laying around, right? But trust me, it does what it does. Um And then to delete a snapshot because I want to get rid of that checkpoint is uh the ability to remove a P FA two group protection group snapshot against our system, but it doesn't actually remove it, it eradicates it.
35:48
So again, if I want to really, really get rid of it, add the eradicate parameter and I'll get a yes, no prompt to actually get rid of that snapshot. So lots of time uh going through different ways to interact with our API and if you start building integrations, uh these are some of the cool things that I like to show customers on how they work. OK.
36:08
So any questions or comments around that team good, you could eradicate that safe. Yeah. Yeah. Yeah. So I mean it kind of could but it's intentionally hard to do, right? So uh who's using the Open Metris Exporter? Anybody you familiar with this project at all?
36:29
Awesome, love it. So we have a github repository where we have this thing called the open Metris exporter, which um allows us to extract that performance data across the API we did a bunch of work in powershell which is cool for discrete management, right? I wanna like find out what's wrong now or maybe over a window span three days ago,
36:47
throw that into a spreadsheet, maybe chart it up with the o metrics exporter does is it takes performance metrics out and sticks them into Prometheus, which is a time series database and then we can build visualizations on top of that in Grafana. And so there's two links here. Uh One's to the official github repository. The other one is to a blog post I wrote about to interact with this.
37:09
But we have uh the integration built in Docker which allows me to stand up all of the infrastructure to have this monitoring stack deployed literally in one line of code. OK? So Docker compose up minus minus detach gets you this the ability to get performance data out of your array into a Grafana dashboard. Is anyone using Grafana in their enterprise? Right?
37:30
Did you know we could do that? We could do this for flash array and flash blade, right? And so this is just a a example dashboard that we've built. But think about all the performance metrics that we talked about today, host volume groups, snapshot sizes, those can all be charted in Grafana super easily.
37:46
And so what we have here is a flash array. We got some interesting information about it. With its version, there's only one array that's cooked up right now. But this could be many, we're getting metrics like throughput, we're getting metrics like data reduction. That's a pretty wild set of data reduction, a re utilization in terms of capacity and then be able to slice and dice across different
38:06
types of performance metrics. So there we have some capacity, we have some IOPS, we also have write and read bandwidth going down further. We even have it down to individual volume level. So some of the things that I've done with this, I'll build unified dashboards across flash array, metrics and SQL server metrics and vmware metrics.
38:25
And so you can have one dashboard that it's all on, right? So imagine that for your operations team, when you're troubleshooting an issue that you can build purpose built dashboards for the SREs for the DB A S for the storage pros to get access to this information. Cool. Yeah. So I again, I'm crazy about performance. So this is where I spend a lot of my time is
38:45
working on integrations around this. So looking at things like uh slow volumes, not on flash ray because they don't get slow. But if you notice we've got some pretty cool metrics there. Uh also being able to look at the host metrics and also uh total array metrics. This is a pretty cool visualization we have. Uh did the team just add it?
39:04
Uh This is unified throughput across multiple arrays, which isn't really great when you're only configured for one array. But trust me, if there's like 10 arrays and you could see like actual total throughput into your storage environment. So how you deal with that data in Grafana? That's totally up to you. This is just a demo that we have um from a dashboard standpoint,
39:23
you can build your own. But think about the core behind this all that perf data that we saw coming out of this power. So male exact same stuff. So you can land this into other platforms like data dog and other enterprise monitoring systems because we're all just talking. Rest, right? Cool.
39:41
Let's jump back to where's the slides? Yeah. Where are the slides? I know I totally lost track of that. Cool. Uh Any questions or comments around the Grafana integration? Is that pretty cool? Yeah. Yeah. So on the dock on the on the um the github
39:58
repository, which I'll pop the links up again in a second. Uh I'm a big Docker fan just and all that infrastructure for Grafana Prometheus and Metris exporter is integrated into there. So you can literally just have Docker compose up and have that up and running. So code dot Pure storage.com is we're kind of the main hub for all of our integrations.
40:19
I spend a lot of time in the open connect world. That's where we have all the SQL server stuff and a lot of this code that we went through today. Uh We also have a community facing slack where you can, we're all kind of hang out and if you have any questions, you can interact with us there, I also have and I realized the text and that's probably a little bit small.
40:38
Uh All of the code for today is available at that github repo, the API reference is there. So if you actually wanna read the docs and not be like Anthony, that's the link uh I blog A ton. And then here are the two links for dope metrics exporter and this deck will be part of the conference uh package. So you don't have to or you can go grab this
40:57
after the fact. So I think we have about four minutes left. Any questions or comments or you wanna hang out with me for the next hour instead of Michael Phelps, didn't think So, so. All right. Well, I'll hang out, uh, for the next few minutes, but if you have any questions, uh, let me know.
41:11
Thank you. Very good.
