Hi,

Is it possible to use GCN3 model to run FS mode? I've done it with VEGA but
I'm interested in GCN3 architecture.

In case it is what would it require?

Thank you.

Regards,
Pau

[Public]

Hi Pau,

It’s probably not possible without a lot of set up, but I have found that GPUs that are “not supported” sometimes still work anyway with certain compute stacks.  You would probably need to:

• Find a BIOS for a GCN3 GPU or rip it from a real GPU – This is so the driver loads so at least this is required
• The GPU must be a dGPU or you would need to write an IOMMU model in gem5 to use an APU
• You may need to use an older software stack – These could be hard to find pre-packaged anymore and might require building from source
• May need to rewrite applications with older software stack API – Dependent on previous step

This will certainly not be easy, even if possible.  Out of curiosity, which parts of GCN3 are you interested in that are different from Vega or Vega does not have?

-Matt

From: Pau Galindo Figuerola via gem5-users gem5-users@gem5.org
Sent: Tuesday, September 19, 2023 4:02 AM
To: gem5-users@gem5.org
Cc: Pau Galindo Figuerola pau.galindo.figuerola@estudiantat.upc.edu
Subject: [gem5-users] GCN3 Full System

Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.

Hi,

Is it possible to use GCN3 model to run FS mode? I've done it with VEGA but I'm interested in GCN3 architecture.

In case it is what would it require?

Thank you.

Regards,
Pau

[AMD Official Use Only - General]

Hi Pau,

You will likely need root access to rip the BIOS from a real GPU.  If you have that, I posted my notes below.  Keep in mind the PCI ids might be different:

“
You can find devices with ROM using find /sys/devices -name rom

For example, my MI210 is PCI device 83:00:

sudo bash -c 'echo 1 > /sys/devices/pci0000:80/0000:80:01.1/0000:81:00.0/0000:82:00.0/0000:83:00.0/rom'
sudo cat /sys/devices/pci0000:80/0000:80:01.1/0000:81:00.0/0000:82:00.0/0000:83:00.0/rom > test.rom
sudo bash -c 'echo 0 > /sys/devices/pci0000:80/0000:80:01.1/0000:81:00.0/0000:82:00.0/0000:83:00.0/rom'
“

In terms of the BIOS causing issues, I am not sure.  If you are running into issues associated with the gfx version, a hacky workaround is to use the following environment variable within gem5 to masquerade as another device, which might be worth a try. For example:

export HSA_OVERRIDE_GFX_VERSION="9.0.2"

-Matt

From: Pau Galindo Figuerola pau.galindo.figuerola@estudiantat.upc.edu
Sent: Friday, April 26, 2024 10:44 AM
To: Poremba, Matthew Matthew.Poremba@amd.com
Cc: The gem5 Users mailing list gem5-users@gem5.org
Subject: Re: [gem5-users] GCN3 Full System

Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.

Hi Matt,

I'm still working on it but I've decided to shift the project towards modeling the AMD V1605B which has vega architecture.
I've the real AMD V1605B device and I'm wondering how could I rip the BIOS from it. Do you have an idea on how to do it?

I build a custom image with a custom ROCm driver from Bruhnspace and I suspect that it is now working because them are intendet to work for the gtx902 gpu, which is the one V1605B has.
Do you think it has something to do with the fact that the BIOS provided in the gem5 repositories is not exactly that one from the V1605B?

Thank you!

Best Regards,
Pau

Missatge de Poremba, Matthew <Matthew.Poremba@amd.commailto:Matthew.Poremba@amd.com> del dia dt., 19 de set. 2023 a les 16:29:

[Public]

Hi Pau,

It’s probably not possible without a lot of set up, but I have found that GPUs that are “not supported” sometimes still work anyway with certain compute stacks.  You would probably need to:

• Find a BIOS for a GCN3 GPU or rip it from a real GPU – This is so the driver loads so at least this is required
• The GPU must be a dGPU or you would need to write an IOMMU model in gem5 to use an APU
• You may need to use an older software stack – These could be hard to find pre-packaged anymore and might require building from source
• May need to rewrite applications with older software stack API – Dependent on previous step

This will certainly not be easy, even if possible.  Out of curiosity, which parts of GCN3 are you interested in that are different from Vega or Vega does not have?

-Matt

From: Pau Galindo Figuerola via gem5-users <gem5-users@gem5.orgmailto:gem5-users@gem5.org>
Sent: Tuesday, September 19, 2023 4:02 AM
To: gem5-users@gem5.orgmailto:gem5-users@gem5.org
Cc: Pau Galindo Figuerola <pau.galindo.figuerola@estudiantat.upc.edumailto:pau.galindo.figuerola@estudiantat.upc.edu>
Subject: [gem5-users] GCN3 Full System

Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.

Hi,

Is it possible to use GCN3 model to run FS mode? I've done it with VEGA but I'm interested in GCN3 architecture.

In case it is what would it require?

Thank you.

Regards,
Pau

[AMD Official Use Only - General]

Hi Pau,

For an APU, I think you can get the bios from the vga rom region in memory.  There is a util file for this in gem5:  util/amdgpu/fs_files/dump_gpu_rom.sh .  However, if you have another GPU such as integrated graphics on the motherboard, it might get that bios instead.

-Matt

From: Pau Galindo Figuerola pau.galindo.figuerola@estudiantat.upc.edu
Sent: Monday, April 29, 2024 12:21 PM
To: Poremba, Matthew Matthew.Poremba@amd.com
Cc: The gem5 Users mailing list gem5-users@gem5.org
Subject: Re: [gem5-users] GCN3 Full System

Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.

Hi Matt,

I cannot find it in the pci devices because the v1605b is an APU that has the GPU embedded. Any idea on where it could be?

By the way export HSA_OVERRIDE_GFX_VERSION="9.0.2" command don't seem to work. The rocminfo and clinfo still detect the GPU as gfx900.
I have tried to modify the gpufs config to make the gfxVersion::gfx902 available from arguments and added the casuistries it to the /src/dev/amdgpu/amdgpu_device.hh and /src/dev/amdgpu/amdgpu_device.cc which already had some code taking into account the gfx902 version.

Thank you!

Best Regards,
Pau

Missatge de Poremba, Matthew <Matthew.Poremba@amd.commailto:Matthew.Poremba@amd.com> del dia dv., 26 d’abr. 2024 a les 19:55:

[AMD Official Use Only - General]

Hi Pau,

You will likely need root access to rip the BIOS from a real GPU.  If you have that, I posted my notes below.  Keep in mind the PCI ids might be different:

“
You can find devices with ROM using find /sys/devices -name rom

For example, my MI210 is PCI device 83:00:

sudo bash -c 'echo 1 > /sys/devices/pci0000:80/0000:80:01.1/0000:81:00.0/0000:82:00.0/0000:83:00.0/rom'
sudo cat /sys/devices/pci0000:80/0000:80:01.1/0000:81:00.0/0000:82:00.0/0000:83:00.0/rom > test.rom
sudo bash -c 'echo 0 > /sys/devices/pci0000:80/0000:80:01.1/0000:81:00.0/0000:82:00.0/0000:83:00.0/rom'
“

In terms of the BIOS causing issues, I am not sure.  If you are running into issues associated with the gfx version, a hacky workaround is to use the following environment variable within gem5 to masquerade as another device, which might be worth a try. For example:

export HSA_OVERRIDE_GFX_VERSION="9.0.2"

-Matt

From: Pau Galindo Figuerola <pau.galindo.figuerola@estudiantat.upc.edumailto:pau.galindo.figuerola@estudiantat.upc.edu>
Sent: Friday, April 26, 2024 10:44 AM
To: Poremba, Matthew <Matthew.Poremba@amd.commailto:Matthew.Poremba@amd.com>
Cc: The gem5 Users mailing list <gem5-users@gem5.orgmailto:gem5-users@gem5.org>
Subject: Re: [gem5-users] GCN3 Full System

Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.

Hi Matt,

I'm still working on it but I've decided to shift the project towards modeling the AMD V1605B which has vega architecture.
I've the real AMD V1605B device and I'm wondering how could I rip the BIOS from it. Do you have an idea on how to do it?

I build a custom image with a custom ROCm driver from Bruhnspace and I suspect that it is now working because them are intendet to work for the gtx902 gpu, which is the one V1605B has.
Do you think it has something to do with the fact that the BIOS provided in the gem5 repositories is not exactly that one from the V1605B?

Thank you!

Best Regards,
Pau

Missatge de Poremba, Matthew <Matthew.Poremba@amd.commailto:Matthew.Poremba@amd.com> del dia dt., 19 de set. 2023 a les 16:29:

[Public]

Hi Pau,

It’s probably not possible without a lot of set up, but I have found that GPUs that are “not supported” sometimes still work anyway with certain compute stacks.  You would probably need to:

• Find a BIOS for a GCN3 GPU or rip it from a real GPU – This is so the driver loads so at least this is required
• The GPU must be a dGPU or you would need to write an IOMMU model in gem5 to use an APU
• You may need to use an older software stack – These could be hard to find pre-packaged anymore and might require building from source
• May need to rewrite applications with older software stack API – Dependent on previous step

This will certainly not be easy, even if possible.  Out of curiosity, which parts of GCN3 are you interested in that are different from Vega or Vega does not have?

-Matt

From: Pau Galindo Figuerola via gem5-users <gem5-users@gem5.orgmailto:gem5-users@gem5.org>
Sent: Tuesday, September 19, 2023 4:02 AM
To: gem5-users@gem5.orgmailto:gem5-users@gem5.org
Cc: Pau Galindo Figuerola <pau.galindo.figuerola@estudiantat.upc.edumailto:pau.galindo.figuerola@estudiantat.upc.edu>
Subject: [gem5-users] GCN3 Full System

Caution: This message originated from an External Source. Use proper caution when opening attachments, clicking links, or responding.

Hi,

Is it possible to use GCN3 model to run FS mode? I've done it with VEGA but I'm interested in GCN3 architecture.

In case it is what would it require?

Thank you.

Regards,
Pau