The colors selected in Setup > Project Preferences > Priorities are used for reporting results and will only show in reports with charts.
The Issue Priority Flag is only Red or Yellow and can't be changed.
Red will be for the highest importance number.
Yellow will be the second highest importance number.
If you Hide a priority then it won't be used in determining the Red/Yellow flag.
If you have 5 priorities set in your system the priority corresponding with the number 5 will be red and the priority number 4 will be yellow. If you were to now add a 6th priority to the list number 6 would be red and number 5 yellow with 4 not having a flag associated.