My latest signs, and Microsoft, I'm. Confusing you that what I
actually useful.
but because of the energy crisis and several other things service shut down.
and because I have some experience with getting computing resources elsewhere, I volunteer to give this seminar to at least give people the it of what is available and start to hit the ball rolling for further discussion.
So
in the server room I have a small workstation cluster.
so test 7. You know it's it's it's a total of about 10 machines. These are the specs, 74 CPU cores and 9 consumer bridge. So, for example, here's a machine in 12 core am the horizon 9,
which all core turbo is a 4 point, one gigahertz, and it's got to Gpus.
Yeah.
yeah, this is nice. And I definitely need it for for the kind of work that I
and since the beginning of this year, after the shutdown, what the it Department has done is, they have installed these devices which measure how much power
is going through the power line. And
Basically, apart from this machine which is in my office.
all of These are in the server room, and the idle powered roll is about.
So if the machines are just sitting there doing absolutely nothing, it still takes about 700.
It's about $82 per machine.
This is not crazy. Given that, you know there is all 90.
If I actually put it under full load. So all Cvs.
If all the Gpus are occupied, it takes about 2.3 people of power.
Okay.
Now I have spoken to the I to the technical department, and they said that for this year the electricity price at this Institute pays is a staggering 357 for it's per kilowatt hour.
So for the people who are joining on zoom, this is roughly €90 cents per kilowatt hour.
Okay, that that's
insert extent that times increase. Yeah, it's a factor of 10 more than what it used to be in sort of excluded this year. Yeah, whatever you can think of, but it's it's very, very much so if I just keep my on throughout the year, and I just keep it idle.
The electricity costs just from that running. Those machines is going to be 2.2 million for it. We're about 5,000 us.
I put it on a full load. It's a factor of 3 and a half higher. So you're looking at something by €18,000 or 7.2,000,400 per year, just for the electricity cost this year Last year it would have been a factor of 10 last, but this year it's from the roof.
Okay.
So the question is.
I kind of feel like, okay, is there something that I can do to to minimize this cost? Given the situation that we are
in?
So one thing that I can do is to look at what kind of high performance, including Ht. High performance computing facilities are available in Hungary.
Right? So you go to the key to the website. They are the organization, the hardware, and most of it is actually very out there. So that, for example, this this is one of the better ones, the debts and 2 which has gpus.
So these K. 20 Gpus. They're about 10 years old right now, 8 to 10 years old. These cpus are similar. I mean when I started in Japan in 2,014, just stack.
We also have the cpus, and they're very slow. Yes.
no, I should not take a differentiate view. It's capable of double precision. Calculation, by your Gpu.
Yes, I understand that.
Yeah, see. But let me finish. I'll get to that. The I have access to these K. 20 is in Japan, and they're, you know, a factor of 3 slower than the hardware that I present.
That's fine.
Okay, Again, in terms of C we use the fastest are about 3. Point 3 gigahertz, but these are also about 5 to 6 years old, so i'm kind of like, and you know I could do this, but I think there must be better options out there. Unfortunately.
in the European Union
there is lots of Hpc. Infrastructure, both at the national level.
which are maintained by respective organizations in each
country, which is a member of grace, which is not necessarily all the EU countries. So this is Keyfood
in Hungary.
and and there are also certain there's also infrastructure which on all time European.
which certain countries are have donated, where members of academia in European Institutes can apply for computer time on these specific resources.
Okay, and that consortium is funded by the European Commission
and is the umbrella term for that. It's it's the partnership for
there. But there
home pages praise dash ri because it is actually a research institute, and it seeks to still update and integrate over
of these computing resources within the
they offer training and coding support
that offer any infrastructure for for administrators. So, for example, the people of that are maintaining these clusters. They can, some of them.
and they also have events like crazy, some this programming guys. And so so I encourage you to check it out.
Prince, as an organization has access to
the national facilities in each country.
So, for example, because I am a in Hungary and all of you are, we can apply to Hungarian facilities
what I cannot apply. For example, for facilities, let's say in the net. I checked this actually yesterday. It is the the entity that maintains this infrastructure Netherlands. And if I wanted to use that, I would need to be based as an academic institution in the same for Germany. It's safe for all the other countries.
Praise has a few Hpc systems based in France, Germany it is, and the blue ones are the ones that have access to graphics.
Okay.
they have free access for Academy academics that are based on the EU. So I can apply for time on these machines and at home.
Okay.
praise as an organization has what is called project access, and they have 2 grounds of applications for here, but deadlines and marginals.
and you will submit an online proposal through this website or
and you're essentially
Now, these are for very large.
massively parallel simulations. These project access, you know. They start at 15 to 30 million
hours per year.
but they also have what they call preparatory access. And this is interesting because it basically gives you access to these facilities to test and debug your
in other words, to make sure that it runs well and efficiently on that hardware which may not always be the same kind of hardware. If you have access to within you. Either your Perspective institute, or national facilities in the country that you're based on
these preparatory access. Calls are always open, and they have porting the access couple of days, the first working day of the quarter. So first March, first of June first
and duration. For all of these project both prep access and the project. Access is always one. So you submit your proposal. It gets peer reviews evaluated. If you win, they say, okay, you get time, and then you have one year from the start to finish to finish your simulations.
We finish early. Good. If you finish late, you're in trouble.
and the kind of hardware that they have is 48 course at 2.7 gigahertz per node.
or they have am the epics a 120 day course per noted 2.5 gigahertz, and they have Nvidia Tesla cards. So these these something 100 cards are the top of the line
computation cards, which cost about €9,000 per card.
and they have the the V 100, which is the previous generation, the current generation 8, 100, and it
the P. 100, which is generation for that.
But these are still very fast. You know jolts that are much faster than the cards that I have in my desktop, although if I,
by a specific trick, I can also get a lot more speed out of them, but I can never compete with these.
I'll get to that. I'll get to that. I can finish.
The other thing that we can apply to is Europe. It is called Euro H. They are also based on the
they use the praise interface for coordination. And so this is the website of these that's very new systems, because this is actually a very new.
So they have a new system in Bulgaria called discover Carolina and Czechia Lumi in Finland, Mexico, and Luxembourg, Vega and Slopedia, which I've been using, and there's a new system being built in Italy, in Bologna. I think it's called
in Spain in Barcelona I forget what it's called. They should be coming online
online proposal Mission again through price calls. And they have also just like phrase, these benchmark. And
so Essentially, you say, okay, I want to test my hardware or my software on your machine, and they have that whole deadline. Why praise every first of the month, except that they get reviewed every once as well.
They have what is called regular access as well with deadlines. 3 cards in here.
and that is a minimum of 10 Cp. 1 million CPU, or 1 million dollars, is the request.
The proposal is in this word. It's 10 pages, Max.
but they require a lot of people info in order for them to get through as much.
Not only do the website, but also myself, recommend
that in order to be able to successfully win one of these
that you first start with a benchmark or a prep access proposal which is starting using.
They give you time a few 1,000 h. Typically, you're on your code. You tell them. Okay, this kind of performance. I get this: how much data I generate the number of files in the amount of traffic, the amount of storage that I need, etc., etc.
You collect that information. You can put that them in your regular access proposal and turn in a nice proposal, and they do it.
They also have something called ex scale access, which is much more, which is similar to the price proposals of 20 million CPU course minimum in in the New Italy facility and 80 million in England Again, it's one year. So from the moment you're
you. You get your access time on the machine. You need to finish your simulations within one year, otherwise they cut you off.
The hardware is always a the epics 128 course, 2.6 gigahertz, which I find
tur goes regularly close to 3 or 2.45, and it has anything yet a 100, which is the state of the art.
or, in the case of Zoom in Finland, which is the largest supercomputer in the EU. It has Amd
graphics cards. Now this is a bit of a bummer, because.
and we have to spend a lot of time developing the language called, which is similar in syntax to c plus.
and they allow you to then write your Gpu program to do whatever you want most of it for all this deep learning and AI stuff. We do the serious science on these cards.
and you can write your code in on the Gpu hardware. Now, Cuba is not compatible with a indeed hardware. You need to use something called, and you can port and via could that code to it. But of course it means you can spend more time developing and debugging your code in order to run it from these pay and instant cards.
So if you're a gpu, you're like me. You use cuda. I cannot recommend you trying to get Gpu access to me because they're paying base cards. Now, I know from a few insiders we're working with am d in building, moving.
That am. B. Is personally helping people report their Kudak code to hit and help them the bucket and so forth.
But I am still hesitating to do that.
It's not a great program.
So this is something to keep in mind
the last kind of call that I will mention, which
is called the collaborative calls also on the praise website, but it's it's essentially the IC ei interactive computing the infrastructure.
They is provided by the phoenix Institute, which is part of the Human Brain Project, also EU funded, based in Spain.
They have systems in France, Germany, Italy, Spain, Switzerland, and they have 3 deadlines a year. They have various services, for for example, computing, but also storage or interactive facilities where you can log into the machine, and you can do things like graphical interfaces. Whatever
minimum request is a modest 5 million hours. The proposals to get our in at this work template is similar to the euro agency, but is much shorter.
4 to 8 pages. They don't require a lot of technical def, so they're the ones that are enough, so he's not so difficult to win.
and you submit it through email
duration is one year again, Amd. And Intel's, even hardware, and and then Vp. 100 cards, which are the pretty 50 iteration, but the 8 100 is not a lot faster.
Their administrative support for these people's experts that has been my experience. I've emailed them to several times, especially one lady in Italy. That's cynic. I she's always very quick to this phone that help me with many things, so you know I can highly recommend
fun fact.
the scientific supervisor at the German use of cluster One here is his own fault, sir, was actually working in a so in a way that is good, because it means that she is in charge
of any proposals that are waiting that are running on this computer. So that kind of makes me feel good that there is actually an expert in my field. Who knows what i'm talking to?
So what has been my experience so far with all of this.
So in 2,022 I was supported 6 and a half 1 million hours and 18.4 million hours this year on user through the I. C. I.
I've only used the Gpu so far, but I don't want to start using CPU later.
and last year I was awarded one and a half 1 billiongpu hours in the Vga cluster, and still really have through your agency. I have submitted another proposal last November, but I don't know what the outcome is, but yet I should be hearing it either later this month, or, in fact.
I was also awarded benchmark access to Vega in Slovenia in 2,021, and Carolina in check, which have very similar.
Yeah, actually.
so
there
the it team can collabor corroborate that I had a lot of time getting access to.
and the reason is that they specifically want you to have a certificate which needs to be issued by you. You're in Hungary in order to be able to locate it to correspond with the free email.
Now the it team spent months trying to get this, and I was kind of like just don't bother, because the hardware is similar to several other costs. And just just don't bother with that sort of thing. So I kind of recommend you apply for.
Go for Vega. Go for
Excuse me. No, it's easy you for that one. And now we can do it.
Okay. But the but the fact that they're making it so.
So the ones that I've used very guy in Slovenia and uses interpreting the use scheduler. Some people will be familiar with it. It's basically the script that you submit to the scheduler which keeps track of all the jobs and then tells you, okay, you're going to run on there for so long and so forth.
It's it's everything but Caroline I use Pbs. Which is a competing system with the different syntax, of course.
So you know, because i'm lazy, I prefer to just use one system, so I prefer to just
and not get confused with all your
they got has excellent technical support. So whenever I email them I get a response to my
or maybe half an hour, and they are to the point. And you know if you're
a power user. Like me to say okay, this this this is my problem. Then you know that's great for them, because they don't exactly
excellent use of the similar.
it's less experience with it. My experience with Caroline that's not good for several reasons, and with their support, was also very good. They were implied. So you know, for them
I have not tried other systems for which the price project proposal. These are massive ones to 20 millionplus
4 h, because they do not meet my needs, but it might be something for you.
I don't know
I have never been in the Gpu for more than 12 h.
and in fact, 90% of the time my Gpu job started by.
So this was a concern. If you mentioned that you might have to wait
in more way. Yes, I have access to our colleagues, Stephanie Burner. Hi, Stephanie, and
there I've had to sometimes wait quite a long time. But on Vega and on you Still, I have never had to wait more than a few hours, 12 h at most. So that's pretty good. But this is for Gpu. I don't know what to do yet.
All awarded projects, I mean there's always a catch with these things right. They allow me to run their hardware, but there's always something that they want for you.
Return, and the apart from the rising proposals, all the Board of projects require a written report at the end no later than 3 months after the
that. This is also true for benchmarks. So if you only get access just to test your code
up to 3 months from the time that the direct your access time expires it when you derive the report. They also have a format for that Microsoft work only.
and you know some of them are more technical than others. But this is, you know the price that we have to pay for using Max and their services.
Of course you should always acknowledge these in public
for that standard, and that's easy, and they deserve that right.
And also, if you're going to apply for time when you think we should not compete, all of them all, all of us in the same system, right? Because they were just going to keep each other waiting
something to think about
limitations.
One thing that I find particularly infuriating, but of course, from a technical point of view it is completely understandable is that you have a limited moment.
So you submit your job. It will run for a some time, which is typically 2 to 7 days, depending on the system, and then it gets killed off right, because then somebody else gets time to those, so that requires frequently starting and stopping and stopping off the simulation
and my simulation, some of which take 2 months to one and a half years. I keep having to restart install every week or so, and eventually you start to accumulate hours.
because the time that you shut down you do a vector dump which is usually in double precision. So you have 16 digits. You'll read that again, and then you continue from there. But over time you accumulate some, some uncertainties in your energy, angular momentum, or whatever.
This is something that
Okay?
Well, you hope
watching.
You are integrated
from all over the beginning
That?
Yeah. So this is something that you have to keep in mind is starting and stopping.
The other thing is, for example, my my cluster here. The CPU is running at about 4 gigahertz
right on all 4 turbo, so I have. Let's say, per machine. I have 12 or some turbo, and I had notes. If I have an in the thread ripper machine with 32,
all of those 4 together, which is amazing.
But most of these, because they're they're in the epics. They have many more cores, but they also generate a lot more which you need to dissipate, so as such, they lower the costume, or most of the servers, because they're thin machines. We could blow a lot of air through them.
So they typically run between 2, that often 3, while my cluster runs a 4 year. So you have a 33, to 40%.
So in my simulation for my appreciation.
Then, if I run it on here, I need to calculate 15 to 17 months.
That's a big difference. So that's something, you know, to keep in mind.
Your project. Duration is here. So if you're since they lower and some of mine do, then I need to stop them back it up and start
a lot of the These systems are oversubscribed. Now, in my case, I Haven't noticed that so much with the euro Hpc. Facilities. But again, these are brand new. So it could be that not many people are aware of the shining you drink that are out there, and therefore they're not so heavily overscribed.
But some of the hardware I've access to in my life because of Stephanie that is oversubscribed, and sometimes I have to wait.
It's too far.
So this is also something you need to factor in when you think okay, I have to here to do my simulation. How much time do I need, and how much time and so forth.
And of course you need to write proposals every year because you have one you access, and you need to write a report.
and if you want access to multiple systems like I
Okay. So this is
what I will say: a a workload overhead that you need to.
I mostly need to do access and gpu access for number crunching.
but it could also be that some of you are like Well, number crunching is not really what I need, but i'm storage right? Because if you're running a discount that's still, and the electricity.
so these I, c. I. E. People. They also offer sport. Now I have no experience with this. I'm. Just mentioning it to you so that you can check it out. You can, you know you can store several terabytes from the services essentially for free
it. That's not your thing. You can also try some commercial options. I just put a few here.
If you want 10 TB storage for a lifetime, you can pay €1,200, which is like half moving forward, or something like that to make it you 2 to the 10 TB storage. Now, I think, for the observers, they're they're probably thinking of 10 TB. Nothing. I don't know for me this. This would be more than I've accumulated over the last 20 years or something.
but maybe a few. It's not, and another one I strike, which Uk: based is about the same price. Just make sure that you don't do it. You don't you store your data in the EU and not in countries like the United States. I don't know.
and also make sure that the Provider doesn't have access to the encryption.
Some of them do that, but not all.
It depends how much privacy you want.
I always recommend people to find those that offer a higher amount
commercial options. So you're like, Well, okay, I don't want to propose all the time. I just want to use infrastructure like Google, Amazon and Microsoft. Because I have all of these, you know, Cloud computing whatever you can do that, too.
So I quickly basically looked at this. Let's say, upon one virtual machine with 30. Yeah. Then, if I use Amazon and I run on their full load for a whole year. 32 cpus occupied. It will cost me about 2.3 million for it
on Microsoft. It's very expensive. I don't know what to them with these prices, but on Google it's about it. It's about 2 million people.
but that only gives me 32 Cpus and I have at least here access to 74.
So if I multiply this by 3, I get about 6 to 7 million hold, which is the same that we pay for the other person.
You're keeping it on.
So I don't win anything by using this commercial
in terms of cost.
Alright, so then the the price and Hero Hpc. Are the
It's so.
Okay, 2 million. So I need to multiply by 2 point something. It's a little cheaper. How much, but you also have to remove the ATM. So it's at least 10. That will
price. Yeah, but not only the concept, maybe. Okay, yeah. But I mean, then, you know, not everybody has a
but it's something to think about if you want
It's
so. I think this was my log. Yeah, that was my last slide. So if you know i'll stop here, and if people have questions, then i'll be happy to talk.
Yes, okay, I have.